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Abstract 

Air  Force  acquisition  programs  invest  a  large  amount  of  resources  to  develop  and 
field  systems.  Many  of  these  resources  go  to  risk  management  in  order  to  help  ensure 
those  programs  finish  successfully  with  respect  to  cost,  schedule,  and  technical 
performance  goals.  The  question  arises  as  to  whether  this  is  a  sound  investment. 
Although  risk  management  has  been  shown  to  be  effective  in  industry,  scant  empirical 
evidence  exists  for  its  effectiveness  within  the  Air  Force.  Correlation  and  regression 
analysis  of  survey  data  show  that  a  positive  relationship  does  exist  between  specific  risk 
practices  and  project  success.  Furthermore,  the  analysis  suggests  that  organizational 
behavior  practices  may  reinforce  that  positive  relationship  in  addition  to  the  structured 
risk  management  steps  prescribed  by  organizations  such  as  the  Department  of  Defense 
and  the  Project  Management  Institute. 
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AN  ASSESSMENT  OF  THE  EFFECTIVENESS  OF  AIR  FORCE  RISK 


MANAGEMENT  PRACTICES  IN  PROGRAM  ACQUISITION  USING  SURVEY 

INSTRUMENT  ANALYSIS 


I.  Introduction 


General  Issue 

The  measure  of  success  for  managing  nearly  any  project  or  program  is  whether  it 
meets  the  given  targets  for  cost,  schedule,  and  performance  (Kerzner,  2013:8;  Mantel  and 
Meredith,  2012:3;  PMI,  2013:35).  Since  projects  are  most  often  executed  under 
conditions  of  uncertainty,  risk  management  aids  the  project  team  in  delivering  a 
successful  project.  Numerous  frameworks  exist  for  project  management  in  general  and 
most  of  those  frameworks  also  include  significant  portions  on  risk  management  (DoD, 
2013;  DoD,  2006;  PMI,  2013,  INCOSE,  2011;  NASA,  2011).  More  applicable  to  an  Air 
Force  audience.  Department  of  Defense  (DoD)  Instruction  5000.02  (DoD,  2013),  the 
overarching  guide  for  defense  acquisition,  contains  numerous  references  to  cost, 
schedule,  performance  and  risk  (DoD,  2013).  What  remains  largely  unknown  is  the 
effectiveness  and  efficiency  of  those  risk  management  efforts. 

Problem  Statement 

The  DoD,  Project  Management  Institute  (PMI),  International  Council  on  Systems 
Engineering  (INCOSE)  and  many  other  organizations  provide  detailed  methods  for  how 
to  approach  risk  management  and  handle  identified  risks  (e.g.,  DoD,  2013;  DoD,  2006; 
PMI,  2013,  INCOSE,  2011;  NASA,  2011).  While  these  methods  might  provide  direction 
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on  how  to  implement  a  risk  management  effort,  they  don’t  offer  a  means  to  evaluate  the 
effectiveness  or  of  that  risk  management  effort.  As  measured  by  the  number  of  and  page 
count  of  its  policies,  the  Air  Force  invests  a  lot  of  resources  in  risk  management  but  the 
question  remains  whether  it’s  effective. 

Research  Objective 

The  purpose  of  this  study  is  to  evaluate  the  effectiveness  of  risk  management 
efforts  within  the  Air  Force  acquisition  community.  By  collecting  survey  information 
from  Air  Force  acquisition  professionals  regarding  whether  and  how  risk  management  is 
actually  practiced,  a  more  accurate  view  of  Air  Force  risk  management  efforts  may  be 
gained.  Ultimately,  this  information  could  be  used  to  determine  not  only  the 
effectiveness  of  risk  management  but  also  provide  insight  into  how  to  better  manage  Air 
Force  projects. 

Investigative  Questions 

Before  exploring  any  additional  questions,  this  study  must  first  find  if  a 
relationship  exists  between  risk  management  efforts  as  a  whole  and  the  project’s 
outcomes  with  respect  to  cost,  schedule,  and  performance.  In  the  event  there  is  no 
relationship,  the  rest  of  the  questions  are  unanswerable. 

Q1 .  Does  a  relationship  exist  between  risk  management  and  an  Air  Force  project 
meeting  its  cost,  schedule,  and  performance  goals? 

Additionally,  the  study  will  identify  how  effective  is  risk  management  when 
applied  to  Air  Force  projects.  Considering  the  importance  placed  on  risk  management  by 
various  project  management  and  system  engineering  frameworks,  it  would  seem 
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practicing  risk  management  in  accordance  with  those  prescriptions  would  be  effective. 
Does  the  presence  of  a  risk  management  effort  lead  to  better  project  outcomes  with 
respect  to  cost,  schedule,  and  performance? 

Q2.  How  effective  is  the  Air  Force  risk  management  effort  in  aiding  its  projects 
to  meet  their  given  cost,  schedule  and  performance  targets? 

The  third  research  question  will  compare  industry  practices  with  Air  Force 
practices.  Risk  management  is  applied  in  both  areas  and  follow  similar  policies  or 
frameworks.  The  expectation,  then,  is  that  the  practices  in  use  should  be  similar.  In  the 
event  they  aren’t,  these  practices  could  be  shared  to  improve  the  success  rates  of  projects 
executed  by  both  groups. 

Q3.  Are  there  differences  between  the  effective  risk  management  practices  used 
in  industry  and  those  used  by  the  Air  Force? 

Hypotheses 

To  better  answer  the  research  questions,  three  hypotheses  will  be  tested.  The  first 
hypothesis  deals  with  the  relationship  that  risk  management  has  with  Air  Force  project 
outcomes.  The  second  question  is  to  determine  if  risk  management  is  effective  in  guiding 
Air  Force  projects  to  meet  their  cost,  schedule,  and  performance  goals.  Finally,  the  third 
hypothesis  focuses  on  whether  or  not  a  difference  exists  between  practices  related  to  cost, 
schedule,  and  performance  from  an  industry  sample  are  the  same  as  those  from  an  Air 
Force  sample.  These  questions  explore  the  effectiveness  of  risk  management,  in  general, 
as  well  as  specific  risk  management  practices  as  they  relate  to  Air  Force  projects  and  how 
this  effectiveness  compare  to  an  industry  sample. 
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Risk  management  expert  Edmund  Conrow  states  that  the  DoD  Risk  Management 
guide  is  “quite  simply  the  best  introductory  document  on  risk  management”  (Conrow, 
2003:  xx).  Conrow’ s  suggestion  might  lead  one  to  believe  the  DoD  performs  risk 
management  well.  Additionally,  the  basic  principles  of  risk  management  have  been  in 
practice  for  centuries.  It  seems  unlikely,  though  not  impossible,  that  people  would 
continue  to  invest  efforts  into  a  set  of  practices  that  failed  to  provide  some  benefit. 
However,  the  DoD  seems  to  have  trouble  implementing  risk  management  well  (GAO, 
2012).  GAO’s  criticism  might  suggest  that  opportunity  for  improvement  exists.  For 
these  reasons,  one  might  expect  there  to  be  a  weak  but  positive  correlation  between  the 
presence  of  a  risk  management  effort  and  a  project  successfully  meeting  its  cost,  schedule 
and  performance  goals. 

HI .  There  exists  a  statistically  significant  correlation  between  risk  management 
practices  and  Air  Force  project  success  relative  to  cost,  schedule  or  performance  goals. 

There  are  many  prescribed  practices  for  performing  risk  management.  Logic 
dictates  that  at  least  some  of  those  practices  might  be  more  successful  than  others. 
Knowing  which  practices  offer  the  greatest  impact  could  be  useful  to  the  program  office 
and  the  Air  Force  to  better  prioritize  activities.  Project  success  is  often  determined  by  a 
variety  of  influences,  some  known  to  the  project  office  in  advance  and  some  not.  Since 
projects  are  unique  and  multi-determined,  it  seems  unlikely  that  all  practices  will  prove  to 
significantly  affect  project  success.  Still,  one  might  expect  at  least  some  of  them  to  show 
a  relationship. 

H2.  At  least  one  risk  management  practice  will  be  shown  to  significantly  affect 
Air  Force  project  success  relative  to  cost,  schedule  or  performance  goals. 
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Based  on  a  review  of  the  literature,  it  seems  there’s  a  great  deal  of  similarity 
between  the  risk  management  frameworks  in  use  within  the  DoD  and  those  in  use  in  the 
private  seetor.  Specifieally,  the  government  frameworks  from  the  DoD,  the  Air  Foree 
and  NASA  will  be  shown  to  be  similar  to  those  offered  by  PMI  and  INCOSE. 
Additionally,  a  GAO  study  revealed  that,  as  of  2006,  86,181  of  the  1,857,004  employees 
who  left  the  DoD  since  2001  work  for  defense  contractors  (GAO,  2008: 10).  This 
includes  2,435  senior  acquisition  officials  (ibid).  It  could  be  expected  that  defense 
contractors  might  adopt  the  risk  practices  of  their  customer  either  explicitly  or  implicitly. 
They  might  adopt  them  explicitly  to  mirror  the  customer  to  give  that  customer  peace  of 
mind  that  they  understand  what  is  being  asked  of  them.  They  might  also  implicitly 
borrow  some  of  those  same  practices  because  they  share  employees  with  similar 
backgrounds  and  experiences.  Recent  survey  work  by  Dr.  Josef  Oehmen  was  conducted 
to  see  how  industry  organizations  practice  risk  management  (Oehmen,  et  al.,  2014). 

H3.  A  difference  does  not  exist  between  the  effective  risk  management  practices 
used  in  industry  and  those  used  by  the  Air  Force. 

Methodology 

The  focus  for  this  effort  is  the  influence  of  risk  management  on  Air  Force 
acquisition  programs  as  seen  through  the  eyes  of  mid-career,  active  duty  and  civilian 
acquisition  professionals.  These  individuals  will  be  contacted  through  a  course  offered 
by  the  Air  Force  Institute  of  Technology’s  (AFIT’s)  School  of  Systems  and  Logistics. 
The  course  is  targeted  at  acquisition  program  managers  but  is  also  attended,  albeit  in 
lower  numbers,  by  professionals  from  other  career  fields  like  engineering,  logistics  and 


5 


contracting.  The  study  relies  on  these  individuals  to  provide  insight  into  the  broader  Air 
Force  acquisition  system. 

A  survey  method  will  be  employed  to  characterize  how  the  student  population 
praetiees  risk  management.  Survey  respondents  will  be  asked  to  provide  answers  on  a 
wide  range  of  aetivities  involved  with  project  risk  management.  The  survey  will  also 
allow  respondents  to  provide  eomments  regarding  their  experiences. 

Survey  respondents  will  be  asked  to  identify  which  risk  management  praetiees 
were  in  use  on  their  projeet  and  what  level  of  sueeess  that  project  achieved  relative  to  its 
original  eost,  sehedule  and  performanee  goals.  Sinee  risk  management  is  as  mueh  about 
behavior  as  quantitative  results,  a  survey  should  provide  better  insight  into  the  projeet  to 
better  compare  risk  praetiees,  or  behaviors,  with  projeet  outcomes.  Results  from  these 
survey  responses  will  be  analyzed  to  determine  descriptive  statistics  of  those  questions  as 
well  as  potential  relationships  between  answers  provided. 

Tests  of  correlation  and  linear  regression  will  be  used  to  answer  the  first  research 
question  on  the  effeetiveness  of  risk  management.  These  same  tools  will  also  be  used  to 
answer  the  second  research  question  regarding  whieh  speeific  practices  influenee  Air 
Force  project  success.  The  third  research  question  will  be  answered  by  comparing  the 
correlation  of  individual  practices  from  two  different  survey  sets;  one  for  industry 
respondents  and  one  from  Air  Foree  respondents. 

Assumptions/Limitations 

There  are  a  number  of  assumptions  required  for  this  study.  One  of  the  key 
assumptions  is  that  survey  respondents  will  answer  honestly.  This  study  relies  on  self- 
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reported  information  so,  in  addition  to  honesty,  the  respondents  must  also  be  aceurately 
informed  about  the  praetices  in  use  on  their  project  as  well  as  the  project’s  status.  Since 
these  responses  are  also  made  at  a  point  in  time  rather  than  at  the  end  of  the  project,  final 
project  success  is  assumed  to  be  the  same  as  the  information  reported  at  that  moment  in 
time.  These  are  necessary  assumptions  are  necessary  but  impossible  to  guarantee. 

This  study  is  limited  by  a  number  of  factors.  As  this  research  will  rely  on  a  self- 
reported,  voluntary  survey  instrument,  the  data  and  results  are  subject  to  the  typical 
limitations  that  accompany  this  type  of  effort.  Specifically,  to  gain  access  to  the  survey, 
individuals  will  first  have  self-selected  to  attend  a  two-week  course  on  project 
management  and,  further,  agreed  to  complete  the  survey.  This  necessarily  introduces 
some  degree  of  bias — ^the  extent  and  impact  of  which  cannot  be  known. 

A  smaller  than  expected  sample  size  could  also  be  a  limitation  to  generalizing  the 
results.  Surveys  often  have  a  low  response  rate.  Compounding  that  challenge  are  Air 
Force  policies  that  inhibit  a  wider  dissemination  of  the  survey.  Even  if  the  results  prove 
interesting,  the  sample  size  will  be  critical  for  interpretation  and  generalization. 

The  reach  of  this  survey  is  also  limited  by  policies  dealing  with  the  use  of  and 
dissemination  of  surveys  in  the  Air  Force.  These  limitations  might  prevent  the  wider 
dissemination  of  the  survey  needed  to  more  accurately  and  completely  characterize  the 
population  of  DoD  acquisition  professionals  as  a  whole. 

Implications 

Insight  from  this  study  could  lead  to  a  number  of  different  outcomes.  If  risk 
management  efforts  are  highly  correlated  with  project  success,  then  better  training  and 
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more  stringent  poliey  on  risk  management  eould  lead  to  improved  likelihood  of 
acquisition  success.  If  it  turns  out  that  only  a  handful  of  specific  risk  practices  are 
correlated  with  project  success,  then  those  practices  could  be  emphasized.  Similarly,  less 
valuable  practices  might  be  given  less  attention  or  omitted  altogether.  Also,  these  results 
could  provide  additional  insight  relative  to  risk  management  practices  in  the  defense 
industry  vis-a-vis  the  private  sector.  Where  there  is  a  critical  difference,  the  Air  Force 
might  implement  changes  to  better  match  successful  industry  practices. 

All  of  the  previous  implications  assume  the  results  show  risk  management  to  be 
effective  at  all.  The  results  from  the  study  may  show  risk  management  to  have  no  or  little 
impact  on  project  success.  If  that’s  the  case,  resources  previously  assigned  to  risk  efforts 
might  be  redirected  elsewhere  for  efficient  use. 
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II.  Literature  Review 


Chapter  Overview 

This  research  effort  seeks  to  build  on  the  existing  body  of  knowledge  of  risk 
management.  A  better  understanding  of  existing  risk-related  research  might  better  shape 
the  focus  and  evaluation  of  current  practices.  Both  risk  management  and  the  underlying 
influences  of  organizational  behavior  are  multi-dimensional  and  often  involve  other 
disciplines.  Consequently,  the  review  will  also  be  wide  ranging,  but  will  better  set  the 
foundation  for  the  current  assessment  of  DoD  practices.  This  chapter  will  review  current 
structured  risk  management  definitions  and  approaches.  Additionally,  a  review  of  current 
risk  management  practices  and  research  will  be  presented.  Finally,  a  review  of  risk 
management  effectiveness  will  be  provided. 

Risk  Management  Definition  and  Overview 

Understanding  risk  management  requires  understanding  a  few  definitions.  In  this 
section,  we  will  define  “risk,”  “uncertainty,”  “risk  management,”  and  provide  a  review  of 
five  risk  management  frameworks.  This  background  will  provide  the  basis  for  answering 
the  study’s  research  questions. 

The  first  definition  we’ll  explore  is  “risk.”  The  Project  Management  Institute 
(PMI)  defines  a  risk  as  “an  uncertain  event  or  condition  that,  if  it  occurs,  has  a  positive  or 
negative  effect  on  one  or  more  project  objectives  such  as  scope,  schedule,  cost,  and 
quality”  (PMI,  2013:309).  The  DoD  defines  risk  as  “a  measure  of  future  uncertainties  in 
achieving  program  performance  goals  and  objectives  within  defined  cost,  schedule  and 
performance  constraints”  (DoD,  2006:1).  Similarly,  the  United  States  Air  Force  (USAF) 
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defines  a  risk  as  “a  future  event  that,  if  it  occurs,  may  cause  a  negative  outcome  or  an 

execution  failure  in  a  program  within  defined  performance,  schedule,  and  cost 

constraints”  (USAF,  2014:83).  NASA  defines  risk  as: 

The  potential  for  performance  shortfalls,  which  may  be  realized  in  the  future,  with 
respect  to  achieving  explicitly  established  and  stated  performance  requirements. 

The  performance  shortfalls  may  be  related  to  institutional  support  for  mission 
execution  or  related  to  any  one  or  more  of  the  following  mission  execution  domains: 
safety,  technical,  cost,  schedule.  (NASA,  2008:7-8). 

INCOSE  borrows  its  definition  from  E.H.  Conrow  in  writing  that  risk: 

“has  been  defined  as  the  likelihood  of  an  event  occurring  coupled  with  a  negative 
consequence  of  the  event  occurring.  In  other  words,  a  risk  is  a  potential  problem  - 
something  to  be  avoided  if  possible,  or  its  likelihood  and/or  consequences  reduced 
if  not  ”  (INCOSE,  2011:216). 

Table  1,  below,  summarizes  the  definitions  of  risk  as  given  by  the  reviewed 
frameworks.  As  noted  above,  the  definitions  share  the  notion  that  risk  is  a  future  event  of 
unknown  likelihood  with  some  impact  on  project  objectives. 

What  should  be  noted  is  that  all  of  these  definitions  of  risk  share  a  few  key 
attributes.  One  is  that  risk  is  in  the  future.  These  documents  all  offer  additional  details 
beyond  that  provided  in  the  table.  Another  shared  attribute  from  those  documents  is  that 
the  risk  has  some  root  cause  which  may  or  may  not  lead  to  the  risk  becoming  realized. 
Finally,  the  risk,  if  realized,  will  have  some  impact  on  project  success.  These  basic 
concepts  are  shared  by  all  the  reviewed  frameworks. 

However,  there  are  some  differences  between  these  frameworks.  Many  of  these 
documents  distinguish  between  a  not- yet  realized  risk  and  an  issue— something  that  has 
already  or  will  certainly  happen  (USAF,  2014;  DoD,  2006).  This  emphasis  does  not  exist 
with  the  others.  Additionally,  only  one  of  these  guides  suggests  that  the  impact  must  be 
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Table  1.  Definitions  of  Risk 


Source 

Definition  of  Risk 

DoD 

“a  measure  of  future  uncertainties  in  achieving  program  performance 
goals  and  objectives  within  defined  cost,  schedule  and  performance 
constraints”  (DoD,  2006:1) 

INCOSE 

“has  been  defined  as  the  likelihood  of  an  event  occurring  coupled  with  a 
negative  consequence  of  the  event  occurring.  In  other  words,  a  risk  is  a 
potential  problem  -  something  to  be  avoided  if  possible,  or  its  likelihood 
and/or  consequences  reduced  if  not”  (INCOSE,  201 1:216) 

NASA 

“The  potential  for  performance  shortfalls,  which  may  be  realized  in  the 
future,  . .  .related  to  any  one  or  more  of  the  following  mission  execution 
domains:  safety,  technical,  cost,  schedule.”  (NASA,  2008:7-8) 

PMI 

“an  uncertain  event  or  condition  that,  if  it  occurs,  has  a  positive  or 
negative  effect  on  one  or  more  project  objectives  such  as  scope,  schedule, 
cost,  and  quality”  (PMI,  2013:309) 

USAF 

“a  future  event  that,  if  it  occurs,  may  cause  a  negative  outcome  or  an 
execution  failure  in  a  program  within  defined  performance,  schedule,  and 
cost  constraints”  (USAF,  2014:83) 

negative  (USAF,  2014).  Interestingly,  PMI  believes  that  a  risk  could  lead  to  a  positive 
impact  (PMI,  2013).  Some,  including  INCOSE,  use  the  word  “opportunity”  to  refer  to  a 
risk  with  a  positive  impact  to  project  objectives  (e.g.,  INCOSE,  201 1 :227).  Recently,  the 
DoD  issued  an  interim  update  to  its  Risk  Management  Guide  for  Defense  Acquisition.  In 
this  version,  the  DoD  has  also  adopted  the  use  of  “opportunity”  as  uncertain  events  “that 
have  the  potential  for  improving  the  program”  (DoD,  2014). 

A  concept  similar  to  risk  is  “uncertainty.”  Much  of  risk  management  might  be 
traced  to  a  doctoral  dissertation  written  by  Frank  H.  Knight  and  published  as  his  book. 
Risk,  Uncertainty,  and  Profit.  In  this  book,  Knight  provides  definitions  of  “risk”  and 
“risk  proper.”  Here,  Knight  provides  some  insight  into  the  distinction  between  risk  and 
uncertainty: 
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The  essential  fact  is  that  “risk”  means  in  some  cases  a  quantity  susceptible  of 
measurement,  while  at  other  times,  it  is  something  distinctly  not  of  this 
character;  and  there  are  far-reaching  and  crucial  differences  in  the  bearing  of 
the  phenomenon  depending  on  which  of  the  two  is  really  present  and  operating. 
There  are  other  ambiguities  in  the  term  “risk”  as  well,  which  will  be  pointed 
out;  but  this  is  the  most  important.  It  will  appear  that  a  measureable 
uncertainty,  or  “risk”  proper,  as  we  shall  use  the  term,  is  so  far  different  from 
an  unmeasurable  one  that  it  is  not  in  effect  an  uncertainty  at  all.  We  shall 
accordingly  restrict  the  term  “uncertainty”  to  cases  of  the  non-  quantitative 
type.  (Knight,  2006:19-20). 

This  distinction  by  Knight  is  seminal  to  risk  management  and  little  has  changed  since  its 
original  publication  in  1921. 

While  the  difference  between  risk  and  uncertainty  was  clear  to  Knight,  some 
difference  exist  on  these  topics  among  the  frameworks  reviewed.  The  DoD  Risk 
Management  Guide  seems  to  make  no  distinction  between  risk  and  uncertainty  while  the 
Air  Force  does.  The  USAF  describes  uncertainty  as  “the  indefiniteness  about  the 
outcome  of  a  situation”  (USAF,  2014:95).  Additionally,  the  USAF  seems  to  use  the 
word  “concern”  nearly  synonymously  with  “uncertainty.”  Their  provided  definition  for 
“concern”  is  a  “potential  future  event  for  which  the  cross-functional  [Life  Cycle  Risk 
Management]  team  does  not  have  sufficient  information  to  quantify  a  likelihood  or 
consequence”  (USAF,  2014:84).  The  definition  provided  by  PMI  echoes  Knight  in 
suggesting  that  risks  are  known  and  refers  to  uncertainty  as  an  “unknown  risk”  (PMI, 
2013:309).  While  there  are  some  similarities  between  risk  and  uncertainty,  the  key 
difference  is  how  one  deals  with  the  two  concepts.  PMI  suggests  that  risks  can  be 
handled  with  a  risk  reserve  where  the  budget  might  be  assigned  based  on,  for  example,  a 
mathematical  expectancy  of  the  risk  (PMI,  2013).  Conversely,  uncertainty  would  be 
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handled  with  management  reserve  which  might  be  simply  a  percentage  of  the  total  budget 
since  no  information  is  available  at  the  beginning  of  the  project  (ibid). 

With  the  definitions  of  risk  and  uncertainty  explored,  the  review  will  now  focus 
on  the  definition  the  reviewed  frameworks  provide  for  “risk  management.”  Table  2, 
below,  provides  a  comparison  of  definitions  for  “risk  management.” 


Table  2.  Risk  Management  Definitions 


Source 

Definition  of  Risk  Management 

DoD 

“the  overarching  process  that  encompasses  identification,  analysis, 
mitigation  planning,  mitigation  plan  implementation,  and  tracking”  (DoD, 
2006:1) 

INCOSE 

“continuous  process  for  systematically  addressing  risk  throughout  the  life 
cycle  of  a  system,  product  or  service”  (INCOSE,  201 1 :2 15) 

NASA 

“set  of  activities  aimed  at  achieving  success  by  proactively  risk-informing 
the  selection  of  decision  alternatives  and  then  managing  the 
implementation  risks  associated  with  the  selected  alternative”  (NASA, 
2008:1) 

PMI 

“processes  of  conducting  risk  management  planning,  identification, 
analysis,  response  planning,  and  controlling  risk  on  a  projecf  ’  (PMI, 
2013:309) 

USAF 

“the  proactive  management  of  future  uncertainties  to  ensure  that  program 
cost,  schedule,  and  performance  objectives  are  achieved  in  every  phase  of  the 
life  cycle”  (USAF,  2014:  p.  83) 

In  addition  to  the  similarity  these  frameworks  share  with  respect  to  the  definition 
of  risk.  Table  2,  above,  also  highlights  a  great  deal  similarity  with  respect  to  the 
definition  of  risk  management. 

Despite  the  similarities  on  definitions,  these  various  sources  provide  for  slightly 
different  structured  approaches  for  risk  management.  A  few  models  start  first  with  risk 
management  planning  to  decide  at  the  project  level  how  risk  management  will  be  carried 
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out  with  respect  to  risk  identification,  analysis  and  implementation.  Then,  all  the  models 
offer  a  series  of  steps  to  identify,  analyze  and  respond  to  the  risk.  NASA,  however,  uses 
slightly  different  terms  to  perform  the  same  basic  actions. 

While  the  approaches  differ  somewhat,  all  of  them  then  offer  some  steps  for 
identifying,  analyzing  and  responding  to  risks  to  reduce  their  likelihood  and/or  impact. 
Each  framework  offers  a  matrix  that  plots  likelihood  against  impact  to  arrive  at  a 
composite  number  that  provides  some  notion  of  total  risk  to  the  project’s  objectives. 

Most  of  the  frameworks  offer  a  five  by  five  matrix  where  there  are  five  levels  of 
likelihood  of  event  occurrence  and  five  levels  to  denote  severity  of  impact.  PMI  offers 
no  pre-defmed  matrix,  instead  leaving  it  to  the  implementing  organization  to  define  the 
scales  for  likelihood  and  impact. 

Another  interesting  note  is  found  with  the  two  industry  frameworks;  ESfCOSE  and 
PMI.  INCOSE  is  focused  on  systems  engineering  where  PMFs  focus  is  on  project 
management.  While  these  represent  two  different  views  of  executing  projects,  they  both 
share  risk  management  as  a  key  part  of  that  execution  and  hold  similar  views  on  risk  and 
risk  management. 

The  next  step  is  to  implement  some  plan  to  reduce  the  likelihood  of  the  risk,  the 
impact  of  the  risk,  or  both.  All  of  the  frameworks  discussed  provide  a  step  for  tracking 
the  identified  risks  and  their  responses.  These  feedback  loops  create  an  iterative  process 
for  risk  management  that  should  be  ongoing  throughout  the  life  of  the  project.  However, 
no  firm  prescription  is  given  on  exactly  which  practice  to  implement  or  the  effectiveness 
of  those  practices  or  the  effectiveness  of  the  risk  management  effort  as  a  whole. 
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Table  3  below  provides  a  comparison  of  the  step-by-step  approach  prescribed  by 


each  of  these  organizations. 


Table  3.  Risk  Management  Framework  Comparison 


DoD 

INCOSE 

NASA 

PMI 

USAF 

Risk  Management 
Planning 

- 

V 

- 

V 

V 

Risk  Identification 

V 

V 

V 

V 

V 

Risk  Analysis 

V 

V 

V 

V 

V 

Risk  Mitigation 

V 

V 

V 

V 

V 

Risk  Tracking 

V 

V 

V 

V 

V 

(DoD,  2006:4;  INCOSE,  2011:218-219;  NASA,  2008:15;  PMI,  2013:309;  USAF,  2014:90) 


What  should  be  noted  from  the  comparison  is  the  great  deal  of  similarity  among 
the  review  frameworks.  Two  of  the  frameworks,  the  DoD  and  NASA,  do  not  have  an 
explicit  step  for  risk  management  planning;  although,  it  could  be  argued  that  the  risk 
management  planning  called  for  in  the  other  frameworks  is  still  implied  by  the  two 
frameworks  that  do  not.  Both  the  DoD  and  NASA  have  a  number  of  other  required 
policies  that  could  be  argued  to  take  the  place  of  the  missing  planning  step  (DoD,  2013; 
NASA,  2010). 

In  terms  of  options  for  dealing  with  risk,  the  consensus  is  that  one  can  accept, 
avoid,  mitigate  (sometimes  referred  to  as  control)  or  transfer  the  risk  (DoD,  2006:18; 
ISO,  201 1 :225;  PMI,  2013:344).  Risk  acceptance  is  merely  accepting  the  likelihood  and 
consequence  as  is.  This  could  make  sense  for  when  the  cost  of  the  risk  response  is 
greater  than  the  risk  impact  or  when  the  likelihood  and/or  impact  is  deemed  to  be 
negligible.  Avoidance  describes  a  situation  where  the  project  team  may  make  a  decision 
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such  that  the  risk  is  no  longer  a  faetor.  Mitigation  (i.e.,  eontrol)  ean  be  used  when  the 
projeet  team  takes  some  aetion  to  reduce  likelihood  and/or  impaet  associated  with  a  given 
risk.  For  example,  wearing  of  a  camouflage  vest  with  armored  plate  eould  be  expeeted  to 
reduee  the  likelihood  of  being  shot  and  reduee  the  consequenee.  Transference  refers  to  a 
situation  where  the  projeet  team  seeks  to  insulate  itself  from  the  risk  impact  through 
another  party.  Insurance  polieies  are  a  classic  example  of  risk  transfer  where  the  eost  of 
the  impact  has  been  shifted  to  the  insuranee  company. 

The  USAF  offers  those  exaet  four  management  strategies  plus  the  option  of 
“monitoring”  as  a  risk  response  (USAF,  2014:102).  The  Air  Foree’s  inclusion  of 
“monitor”  describes  a  situation  of  short-term  acceptance  but  leaving  open  the  opportunity 
to  take  some  action  later.  Note  that  this  is  a  distinet  strategy  as  opposed  to  simply 
tracking  the  risk.  These  options  provide  eategories  of  ways  to  reduee  the  likelihood  or 
impaet  of  a  given  risk  (with  the  exception  of  acceptanee)  but  they  don’t  provide  a 
measure  of  effeetiveness. 

One  interesting  omission  from  these  frameworks  is  that  none  has  a  means  of 
tracking  the  effeetiveness  of  the  risk  management  effort.  Many  have  steps  to  track  the 
implementation  of  a  given  risk  response,  but  that  fails  to  fully  address  the  implementation 
of  risk  management  on  the  whole  or  provide  some  insight  in  advanee  as  to  whieh  praetiee 
might  be  most  likely  to  improve  the  project’s  chance  of  suecess.  INCOSE  ealls  for  the 
project  team  to  “evaluate  the  risk  management  process”  but  this  is  foeused  at  the  level  of 
a  given  risk  much  like  how  the  other  frameworks  call  for  “tracking.”  None  of  the 
frameworks  offer  that  empirical  insight  to  practitioners  to  better  manage  their  projects. 
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With  a  working  set  of  definitions  of  “risk”  and  “risk  management”  in  hand,  the 
study  will  now  provide  a  review  of  risk  management  practices.  The  focus  will  now  turn 
to  a  review  of  risk  management  practices  in  an  effort  to  provide  the  groundwork  for 
answering  the  research  questions  for  this  study.  Recall  that  the  first  question  asks  if  risk 
management  practices  have  a  relationship  with  project  success  relative  to  cost,  schedule, 
and  technical  performance  and  the  second  research  question  asks,  assuming  a  relationship 
exists,  if  risk  management  if  effective.  A  review  of  relevant  research  is  provided  to 
explore  these  questions. 

Risk  Management  Practices 

Some  of  the  risk  responses  discussed  above  could  be  considered  practices  in  risk 
management,  but  there  are  many  others.  Oehmen  et  al.  recently  structured  a  survey 
asking  95  questions  (of  a  survey  total  of  173)  on  specific  risk  management  practices 
(Oehmen  et  ah,  2014).  In  that  survey,  Oehmen,  et  ah,  documented  six  categories  of  risk 
management  practices,  or  categories  of  practices,  and  mapped  them  to  risk  management 
process  steps.  The  categories  resulted  from  the  author’s  use  of  a  factor  analysis  to  reduce 
the  number  of  practices.  Those  basic  steps  are  based  on  their  analysis  of  risk 
management  frameworks  reviewed  above  with  the  addition  of  the  ISO  31000  standard  on 
risk  management.  Table  4,  below,  is  reproduced  from  that  article  (Oehmen,  et  ah, 
2014:447)  with  permission  from  the  authors: 

The  first  category  deals  with  the  skills  and  resources  required  to  perform  risk 
management  activities.  The  second  category  is  aimed  at  developing  a  risk  management 
approach  tailored  to  the  unique  needs  of  the  given  project.  Neither  of  these  map  directly 
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Table  4.  Risk  Categories 


Risk  Management 
Categories 

Risk  Ident. 

Risk 

Analysis 

Risk 

Eval. 

Risk 

Treat. 

Monitor 

and 

Review 

other 

1 .  Develop  risk 
management  skills  and 
resourees 

X 

2.  Tailor  risk 
management  to  and 
integrate  it  with  new 
produet  development 

X 

3.  Quantify  the  impaet 
of  risks  on  your  main 
objeetives 

X 

4.  Support  all  eritieal 
deeisions  with  risk 
management  results 

X 

X 

5.  Monitor  and  review 
your  risks,  risk 
mitigation  aetions,  and 
risk  management 
proeess 

X 

X 

6.  Create  transpareney 
regarding  new  produet 
development  risks 

X 

(Oehmen,  et  al.,  2014:447) 


to  a  risk  management  step  in  the  frameworks  reviewed  but,  at  least  in  the  opinion  of  PMI, 
are  taken  as  given  prior  to  the  beginning  of  a  project  (PMI,  2013).  Additionally,  the  DoD 
also  assumes  that  the  requisite  resources  are  in  place  to  manage  the  project  successfully 
(DoD,  2013). 

The  third  category  deals  with  how  well  the  risks  are  analyzed.  In  the  previous 
frameworks,  the  organizations  implementing  the  project  will  have  identified  the 
likelihood  and  impact  of  the  given  risks.  Organizations  can  evaluate  the  risk  with  respect 
to  cost,  schedule,  performance  or  a  variety  of  other  factors.  For  instance,  some 
organizations  might  consider  the  ability  to  commercialize  the  product  to  be  a  key  factor 
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for  evaluation.  Some  organizations  will  evaluate  risks  by  rank  ordering  (e.g.,  1  to  10)  or, 
in  the  case  of  more  rigorous  approaches,  the  risks  can  be  evaluated  more  probabilistically 
using  distributions  for  likelihood  and  impact.  Statistical  methods  like  Monte  Carlo  can 
more  comprehensively  evaluate  the  cost  and  schedule  risk  to  the  project  and  provide 
more  insight  that  evaluation  approaches  that  provide  a  single  number  for  likelihood  and  a 
single  number  for  impact.  Where  cost  and  schedule  are  a  function  of  each  other,  this 
might  give  a  more  complete  picture  of  the  project’s  risk. 

The  fourth  risk  management  category  has  to  do  with  how  well  the  risks  are 
evaluated  and  how  they  are  treated  to  make  the  best  trade-off  decisions  within  the  project. 
Risk  evaluation  could  influence  the  requirements  of  the  project  as  well  as  resource 
allocation  within  the  project.  Trade-off  studies  or  decisions  made  by  weighing  costs  and 
benefits  could  be  other  forms  of  evaluation.  The  second  set  of  practices  within  this 
category  has  to  do  with  how  risks  are  treated  to  reduce  likelihood  and/or  impact.  This  is 
where  most  of  the  practices  identified  by  Oehmen,  et  al.  are  located. 

Broadly,  Oehmen,  et  al.  consider  that  risk  mitigation  may  be  carried  out  to 
improve  organizational  efficiency,  project  management  efficiency,  improve  requirements 
selection,  or  reduce  technological  risk  (Oehmen,  et  al,  2014).  They  consider  the  use  of 
financial  or  schedule  reserves  to  improve  organizational  efficiency.  To  make  use  of  a 
risk  reserve,  the  organization  would  set  aside  an  additional  amount  of  money  or  time  to 
deal  with  the  unexpected.  These  are  sometimes  referred  to  as  management  or  risk 
reserves. 

The  selection  of  contract  type  is  another  way  they  have  identified  to  improve 
organizational  efficiency.  Selection  of  contract  type  can  be  an  important  consideration  in 
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allocating  risk  between  customer  and  developer.  A  fixed-price  contract  is  expected  to 
shield  the  customer  from  cost  overruns  and  place  more  of  the  development  risk  on  the 
provider.  Conversely,  a  cost-reimbursable  contract  makes  the  customer  responsible  for 
cost  overruns  and  largely  shields  the  developer  from  risk.  This  is  a  common  decision 
within  the  DoD  that  has  implications  for  how  the  contractor  is  incentivized  to  perform  the 
requested  service  or  product  development.  With  a  fixed-price  contract,  the  contractor  is 
required  to  deliver  what  was  promised  in  the  contract  and  is  held  financially  responsible 
for  cost  overruns.  Under  a  cost-reimbursable  contract,  the  contractor  is  obliged  only  to 
deliver  “best  effort,”  and  cost  overruns  are  paid  for  by  the  customer. 

Project  management  efficiency  practices  seek  to  improve  the  standardization  and 
quality  of  process  in  use  by  the  project  team.  These  would  include  additional  cost, 
schedule  and  performance  studies  as  well  as  process  monitoring  and  training  programs. 
The  original  set  of  requirements  will  begin  to  dictate  the  risk  profile  for  the  project;  the 
more  ambitious  the  requirement,  the  greater  risk  of  complications.  Within  the  DoD, 
Technology  Readiness  Levels  (TRLs)  are  one  method  of  measuring  how  ready  a  given 
technology  is  to  be  put  into  use  for  a  given  project.  Selecting  only  ready  or  nearly  ready 
technology  is  expected  to  reduce  the  project’s  risk. 

The  fifth  category  has  to  do  with  how  the  risks  are  monitored  and  reviewed.  All 
of  the  frameworks  discussed  include  a  loop  to  re-evaluate  the  already  identified  risks  as 
well  as  continuously  look  for  new  ones.  Organizations  can  choose  to  do  this  at  given 
intervals  (e.g.,  daily,  weekly,  monthly)  or  in  response  to  specific  events  like  significant 
milestones  or  prior  to  key  decision  points.  Organizations  can  also  choose  the  depth  of 
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risk  review.  For  example,  some  organizations  might  choose  to  review  only  the  top  ten 
risks  or  only  those  they  consider  critical. 

The  sixth  and  final  category  includes  methods  to  ensure  transparency  of  reporting 
with  respect  to  risk  management.  Where  the  goal  of  risk  management  is  to  reduce 
likelihood  and/or  impact  of  risks,  more  transparency  should  aid  in  good  decision  making. 

While  there  are  some  differences  between  the  frameworks,  this  study  will  use  a 
common  set  of  steps  and  terms.  Borrowing  from  the  most  expansive  frameworks,  this 
study  will  acknowledge  the  following  risk  management  steps:  risk  management  planning, 
risk  identification,  risk  analysis,  risk  mitigation  and  implementation,  and  risk  tracking. 
Additionally,  while  most  of  the  frameworks  agree  that  risk  may  be  positive  or  negative, 
this  study  will  use  “risk”  when  referring  to  a  potential  negative  outcome  and 
“opportunity”  when  referring  to  a  positive  outcome.  This  should  add  clarity  for  the 
reader. 

Effectiveness 

There  appears  to  be  a  great  deal  of  similarity  in  the  risk  management  approaches 
surveyed  above.  However,  these  details  still  fail  to  answer  the  second  research  question; 
that  is,  whether  risk  management  is  effective  at  improving  project  success.  A  significant 
challenge  exists  in  providing  empirical  evidence  of  risk  management  effectiveness;  every 
project  is  unique.  Indeed,  a  recent  study  notes,  “the  literature  does  not  document  any 
empirical  investigations  of  how  risk  management  activities  can  be  effectively  and 
efficiently  implemented”  (Oehmen  et  ah,  2014:443).  This  project  uniqueness  makes 
developing  a  valid  control  group  difficult  if  not  impossible. 
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Still,  some  studies  have  provided  some  insight  into  the  effectiveness  of  risk 
management.  One  recent  study  that  evaluated  the  effectiveness  of  risk  management  was 
completed  by  Elm  in  2012  as  part  of  a  larger  effect  to  ascertain  the  overall  effectiveness 
of  Systems  Engineering  (Elm,  J.P.  and  Goldnerson,  D.R.,  2012).  This  study  was 
performed  as  a  collaboration  between  the  National  Defense  Industrial  Association  System 
Engineering  division  (NDIA-SED),  the  Institute  of  Electrical  and  Electronic  Engineers 
Aerospace  and  Electronic  Systems  Society  (IEEE- AES S),  and  Carnegie  Mellon’s 
Software  Engineering  Institute  (SEI).  Elm’s  study  looked  at  a  number  of  systems 
engineering  practices  including  risk  management. 

Looking  at  the  broad  implementation  of  risk  management,  this  study  delivered  a 
Goodman-Kruskal’s  Gamma  of  +0.21,  suggesting  a  moderately  positive  correlation 
between  the  implementation  of  risk  management  and  project  performance  (ibid:  p.  xii). 

In  this  study,  project  performance  was  measured  by  degree  of  adherence  to  assigned  cost, 
schedule  and  technical  performance  goals.  Systems  engineering  implementation  was 
measured  by  surveying  respondents’  views  on  the  implementation  of  work  products 
prescribed  by  the  Capability  Maturity  Model  -  Integrated  (CMMI).  More  specifically, 
the  study  looked  at  the  impact  on  “lower  challenge’’  and  “higher  challenge’’  projects  and 
found  gamma  values  of  +0. 1 8  and  +0.24  respectively  (ibid:76).  This  has  a  number  of 
possible  explanations.  It  could  suggest  that  risk  management  becomes  more  effective  as 
project  challenge  increases.  It  could  also  suggest,  as  noted  in  another  study  (DoD,  2014), 
that  project  practices  change  based  on  the  amount  of  risk;  that  is,  more  effort  may  be 
placed  on  higher  risk  projects. 
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The  DoD  routinely  reviews  acquisition  performance,  which  includes  some 
measure  of  risk  management  practice  effectiveness.  These  reviews  are  documented  in 
GAO  reports,  Select  Acquisition  Reports  and  a  various  other  means.  Fixed-price 
contracts  are  largely  regarded  as  a  method  to  protect  the  buyer  from  cost  overruns  where 
cost  reimbursable  contracts  shift  the  cost  risk  from  the  seller  to  the  buyer.  For  this 
reason,  recent  DoD  policy  has  stated  a  preference  for  use  of  fixed-price  contracts  (DoD, 
2013),  but  the  2014  annual  report  on  the  Performance  of  the  Defense  Acquisition  System 
found  “no  statistical  correlation  in  either  development  or  early  production  between 
performance  (cost  or  schedule  growth)  and  broad  contract  type  (fixed-price  and  cost- 
plus)”  (DoD,  2014:87).  This  analysis  covers  433  major  defense  acquisition  programs 
(MDAPs)  from  1970  to  201 1  (ibid).  However,  the  report  also  discusses  the  difference  in 
contract  type  and  cost  performance  and  that  this  distinction  between  contract  types  is  “not 
the  divide  on  effectiveness  ”  [emphasis  in  the  original]  (DoD,  2014: 101).  Instead,  the 
difference  is  that  fixed-price  contracts  should  be  used  in  low  technical  risk  situations  and 
cost  reimbursable  contracts  are  most  appropriate  in  higher  technical  risk  circumstances. 
This  also  highlights  again  the  difficulty  in  establishing  a  true  control  group  for  measuring 
risk  management  effectiveness  since  it  would  be  unreasonable  to  assign  different  contract 
types  to  projects  of  the  same  risk  level. 

The  survey  used  by  Oehmen,  et  al  (hereafter  simply  referred  to  as  the  Oehmen 
study)  looked  to  identify  specific  practices  that  improved  either  project  or  product 
success.  The  survey  from  that  research  effort  discovered  a  total  of  30  specific  risk 
management  practices  that  were  positively  associated  with  program  outcomes  (Oehmen, 
et  ah,  2014).  There  were  five  practices  noted  for  each  of  the  first  three  categories  in 
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Table  4  above;  three  for  category  4;  nine  for  category  5;  and  three  for  category  6.  These 
specific  practices  were  also  tested  for  Goodman-Kruskal  Gamma  associations  between 
these  practices  and  a  number  of  outcomes  including  project  and  product  success  (ibid). 
What’s  interesting  about  these  findings  is  that  they  validate  many  of  the  risk  management 
frameworks  above.  Many  of  the  practices  are  already  encouraged  by  those  frameworks. 
The  ten  practices  found  to  be  successful,  but  not  called  for  explicitly  by  the  frameworks, 
are  categorized  in  Table  4  above  under  “other”  and  deal  primarily  with  organizational 
behavior  and  other  management  factors. 

The  Oehmen  study  used  four  outcome  variables  which  were  mapped  to  two 
success  parameters.  How  well  the  project  met  its  cost  target  and  how  well  the  project 
met  its  schedule  target  were  two  of  the  outcome  variables,  each  of  which  was  mapped  to 
project  success.  The  two  other  outcome  variables,  technical  performance  success  and 
customer  satisfaction  success,  were  considered  by  Oehmen,  et  al.  to  represent  product 
success. 

Their  survey  found  ten  practices  to  be  positively  correlated  with  success,  eight  for 
product  success  and  two  for  project  success.  Table  5  lists  the  variables  Oehmen,  et  al. 
found  to  be  statistically  significantly  correlated  with  outcome  variables  that,  based  on  the 
research  questions  for  this  study,  are  most  likely  to  be  relevant.  Each  of  the  variables  has 
a  Kruskal  Goodman  Gamma  that  ranges  from  0.41  to  0.52  which  represents  a  moderate  to 
strong  relationship  between  these  independent  variables  and  the  two  dependent  variables 
(schedule  success  and  customer  satisfaction  success). 
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Table  5.  Significant  Schedule  and  Customer  Satisfaction  Correlations 


Schedule  Success 

Customer 

Satisfaction  Success 
(Kruskal-Goodman 
Gamma) 

Variable 

(Kruskal-Goodman 

Gamma) 

1 . 1  Our  employees  are  motivated  to 
perform/implement  risk  management(RM) 

- 

0.43 

1.5  Our  RM  takes  human  and  cultural 
factors  into  account 

- 

0.41 

2.5  The  RM  process  is  effectively 
integrated. . . 

- 

0.46 

3.1  The  impacts  of  risk  are  quantified 
using  cost  as  a  dimension 

0.41 

- 

3.2  The  impacts  of  risk  are  quantified 
using  technical  performance. .  .as  a 
dimension 

0.49 

- 

4. 1  Resources  are  allocated  to  reduce 
largest  risks  as  early  as  possible 

- 

0.42 

5.8  Our  RM  is  dynamic,  iterative  and 
responsive  to  change 

- 

0.40 

5.9  Our  RM  is  systematic,  structured  and 
timely 

- 

0.42 

6.2  Our  RM  is  transparent  and  inclusive 
towards  all  stakeholders 

- 

0.51 

6.3  Our  RM  explicitly  addresses 
uncertainty 

- 

0.42 

(Oehmen,  et.  ah,  2014) 


Of  the  eight  practices  associated  with  product  success,  two  were  considered  to  be 
a  part  of  the  “skills  and  resources”  category  from  the  Oehmen  study.  Those  two 
measured  the  motivation  of  employees  to  perform  risk  management  and  whether  the  risk 
management  effort  considers  human  or  cultural  factors.  Risk  management  process 
integration  and  resource  allocation  to  reduce  risk  were  also  found  to  be  positively 
correlated  with  product  success.  The  final  four  practices  measured  how  responsive  and 
structured  the  risk  effort  was  and  how  transparent  the  effort  was  as  well  as  how  well  the 
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risk  effort  addressed  uneertainty.  All  eight  of  these  variables  were  reported  having  a 
Goodman-Kruskal  Gamma  of  between  0.40  and  0.51,  indicating  a  good  correlation 
between  these  practices  and  their  associated  outcomes.  These  findings  suggest  not  only 
that  risk  management  may  be  correlated  with  product  success,  but  also  that  specific 
practices  might  be  related  to  project  success. 

The  Oehmen  study  also  considered  additional  outcome  categories  (quality 
decision  making,  high  program  stability,  and  a  proactive  and  open  organization)  but  were 
not  considered  relevant  for  this  study’s  research  questions  since  they  do  not  relate  to  the 
definitions  of  project  success  as  defined  by  the  reviewed  frameworks. 

None  of  their  variables  were  found  to  be  correlated  in  a  statistically  significant 
manner  with  either  cost  success  or  technical  performance  success.  However,  the 
variables  reported  do  offer  a  moderate  covariance  with  the  outcomes.  This  offers  some 
indication  that  risk  management  is  at  least  correlated  with  some  measures  of  project 
success. 

Air  Force  and  Industry  Risk  Management  Approaches 

The  third  research  question  asks  if  there  are  difference  between  practices  in  use  in 
industry  and  practices  in  use  on  Air  Force  projects.  Based  on  the  information  contained 
in  Table  1  and  Table  3,  it  appears  that  DoD  and  Air  Force  guidance  is  quite  similar  in 
policy  to  the  prescriptions  provided  by  leading  industry  groups  such  as  PMI  or  INCOSE. 
Assuming  the  DoD  policy  is  implemented  as  written,  one  could  reasonably  expect  the 
RM  approaches  used  in  both  the  Air  Force  and  the  private  sector  to  be  quite  similar. 

What  remains  to  be  seen,  however,  is  whether  the  approaches,  or  practices  in  use,  are 
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similar.  A  comparison  of  risk  management  practiees  in  use  in  the  Air  Force  is  explored 
next. 

Risk  Management  in  Practice  within  the  Air  Force 

A  comprehensive  review  of  all  AF  programs  is  beyond  the  scope  of  this  study  but 
two  programs  may  make  for  a  useful  comparison.  Both  of  these  implemented,  or  failed 
to  implemented,  some  of  the  practices  discussed  previously.  They  are  the  F- 16  and  F-35 
fighter  programs. 

The  F-16  was  bom  out  of  the  Lightweight  Fighter  Program  (LFP).  This  effort 
began  in  earnest  in  1972  and  sought  a  small,  inexpensive,  and  maneuverable  fighter  that 
would  be  less  expensive  to  develop  and  provide  the  “low”  part  of  the  high-low  fighter 
mix  the  Air  Force  was  seeking  (Aronstein  &  Piccirillo,  1996).  The  “high”  role  would  be 
filled  by  the  F- 15;  already  in  development  and  production  at  the  time. 

The  AF  selected  two  aircraft  for  test  demonstration:  the  YF-16,  developed  by 
General  Dynamics,  and  the  YF-17,  developed  by  Northrop.  The  aircraft  were  selected 
not  based  on  their  expected  performance  but  on  their  dissimilarity.  There  was  no 
expectation  to  purchase  either  aircraft.  Instead,  the  design  features  were  expected  to  be 
combined  in  some  future  aircraft.  Selection  of  two  designs  for  the  same  purpose  is 
sometimes  referred  to  as  competitive  prototyping.  This  is  expected  to  reduce  production 
risk  by  spending  more  money  in  development.  Two  designs  offer  a  greater  opportunity 
of  finding  at  least  one  desirable  aircraft  and  the  presence  of  competition  might  be 
expected  to  create  downward  price  pressure  for  the  next  phase. 
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The  contract  used  for  this  development  was  a  Cost  Plus  Fixed-Fee  (CPFF).  With 


this  contract  type,  the  contractor  is  paid  a  fixed  fee  and  the  customer  (government) 
assumes  the  risk  for  any  cost  overruns.  Additionally,  the  developer  is  only  obliged  to 
deliver  their  “best  effort.”  In  this  case,  the  technical  or  performance  risk  is  home. 
Because  of  that,  the  developer  may  be  willing  to  accept  more  technical  risk  which,  while 
it  might  lead  to  a  poor  outcome,  may  also  lead  to  a  higher  performance  aircraft.  The 
developer  in  this  instance,  General  Dynamics,  felt  comfortable  pursuing  the  technical  risk 
associated  with  a  fly-by- wire  flight  control  system  in  the  YF-16.  This  flight  control, 
while  revolutionary  at  the  time,  would  allow  the  aircraft  to  be  more  maneuverable  but 
would  also  require  more  computer  control  of  the  flight  control  surfaces.  Despite  the 
contract  type,  Northrop  still  did  not  feel  comfortable  with  a  fly-by-wire  flight  control  for 
the  YF-17. 

There  was  also  no  scheduled  delivery  date.  Instead,  the  Air  Force  wanted  the 
contractor’s  very  best  effort.  Former  Air  Force  Systems  Command  commander  General 
Alton  Slay  suggested  that  the  individual  aircraft  testing  will  take  place  whenever  they’re 
delivered.  This  freedom  to  deliver  should  allow  the  contractor  to  take  more  design  risk 
which  might  be  expected  to  lead  to  greater  performance  if  one  were  willing  to  pay  for  the 
added  cost  and  schedule  impacts  should  they  occur.  This  again  highlights  the 
interdependence  between  cost,  schedule  and  performance. 

The  request  for  proposal  for  the  LFP  was  quite  restrained  coming  in  at  only 
twenty- five  pages  (Ward,  2009).  Relative  to  the  F-15,  the  LFP  sought  a  restrained 
technical  solution  well  within  the  state  of  the  art  for  the  time  (ibid).  This  reinforces  the 
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DoD  preference  for  more  proven  technology  (as  measured  by  TRLs)  for  use  in 
acquisition  programs. 

Another  interesting  feature  of  the  F-16  is  its  longevity  and  the  number  of 
successful  modifications  and  upgrades.  Originally,  however,  there  was  no  immediate 
plan  to  upgrade  the  aircraft.  Recent,  though  not  current,  DoD  policy  called  for 
evolutionary  development  where  the  product  would  be  planned  for  an  initial  production 
run  with  planned  increments  for  improvement  (DoD,  2008).  While  it’s  unclear  why  this 
preference  was  not  included  in  current  policies,  this  preference  was  expected  to  field 
capability  sooner  and  less  expensively  than  the  alternate  approach,  i.e.,  single-step  to 
maturity.  In  this  case,  the  product  is  kept  in  development  until  the  product  meets  100% 
of  all  desired  requirements.  In  hindsight,  the  F-16  appears  to  employ  evolutionary 
development. 

Only  during  test  did  a  confluence  of  events  take  place  where  the  Air  Force 
selected  the  YF-16  as  the  “winner.”  Eighteen  months  later,  the  F-16  was  in  production. 
While  not  immediately  selected,  the  YF-17  was  the  basis  for  what  became  the  F-18,  a 
more  than  capable  aircraft.  In  this  instance,  the  application  of  these  risk  practices  made 
for  a  highly  successful  effort  for  both  tested  aircraft. 

In  contrast,  the  more  recent  F-35  development  has  proved  to  be  a  different 
program.  Rather  than  a  small,  inexpensive,  and  limited  design,  the  F-35  is  actually  three 
different  aircraft  (F-35A,  F-35B,  and  F-35C)  with  substantially  different  features.  While 
they  are  similar  from  the  outside,  the  Air  Force  model  (F-35A)  is  different  from  the  Navy 
version  (F-35C).  In  addition  to  other  features,  the  F-35C  has  a  larger  wing  area  and 
different  landing  gear.  The  model  developed  for  the  Marines  (F-35B)  is  the  most 
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dissimilar  in  that  it  features  a  completely  different  engine  that  allows  it  to  perform  take¬ 
offs  from  short  runways  and  to  land  vertically.  This  feature  is  also  referred  to  as  Short 
Take-Off  and  Vertical  Landing.  Clearly,  the  program  is  not  developing  one  aircraft  but 
rather  three  distinct  and  different  aircraft.  This  is  a  much  greater  technical  challenge  than 
the  more  limited  requirements  provided  for  the  F- 16. 

Contrary  to  the  F-16’s  appearance  of  evolutionary  acquisition,  the  F-35  program’s 
approach  seems  to  use  the  opposite  paradigm:  single-step  to  maturity.  Previously,  the 
single-step  to  maturity  approach  was  expected  to  keep  the  system  in  development  longer, 
resulting  in  greater  costs  and  greater  delay  in  delivering  the  product  or  service  to  the  end 
user. 

The  approach  to  engineering  development  is  also  different  between  the  F-16  and 
F-35.  Starting  the  in  the  1950’s,  the  DoD  began  pursuing  concurrent  development  of 
individual  sub-systems.  That  is,  designers  would  develop  an  aircraft’s  airframe  at  the 
same  time  as  they  developed  the  engine  at  the  same  time  they  would  develop  other  sub¬ 
systems.  This  is  in  contrast  to  the  previous  approach  where  individual  items  were 
developed  sequentially.  While  this  took  more  time,  it  reduced  the  risk  of  integration. 

This  is  the  time  in  the  program  where  sub-systems  are  matched  together.  Concurrent 
development  sometimes  resulted  in  re-work  to  address  conflicts  from  concurrent 
development. 

With  the  F-35  program,  the  DoD  has  decided  to  pursue  not  the  concurrent 
development  of  individual  sub-systems  but  rather  they  have  opted  to  overlap  whole 
phases  of  the  defense  acquisition  system.  DoD  policy  calls  for  technology  development 
to  mature  risk  to  an  acceptable  level.  Notionally,  this  technical  risk  is  measured  by 
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Technology  Readiness  Levels  (TRLs)  where  a  given  technology  should  be  at  TRL  6  prior 
to  entering  the  Engineering  and  Development  Phase  (EMD)  (DoD,  2011).  Then, 
engineering  development  can  be  performed  to  design  and  develop  a  production-ready 
system.  Finally,  the  system  is  produced  and  operated.  The  F-35  program  has  opted  to 
perform  technology  development  at  the  same  time  as  engineering  development  at  the 
same  time  as  production.  This  leads  to  production  items  which  are  already  obsolete 
before  they  leave  the  production  line.  This  practice  also  defeats  the  cost  efficiency  gains 
available  from  producing  the  same  item  multiple  times.  In  effect,  each  aircraft  is  unique. 
The  military  should  have  learned  its  lessons  from  previous  acquisition  failures  brought 
about  by  concurrent  development  (Converse,  2012)  but,  if  it  did,  those  lessons  were  not 
applied  to  the  F-35. 

This  study  seeks  to  evaluate  the  effectiveness  of  risk  management.  However,  it’s 
plain  that  the  F-35  has  deviated  from  what  many  of  the  previous  frameworks  define  as 
best  practices  in  risk  management.  Interestingly,  the  F-35  seems  to  have  violated 
notional  DoD  policy  that  suggests  a  design  should  include  proven  technology  and  be 
proven  through  testing  before  going  into  production  (DoD,  2013).  The  F-35  has  called 
into  question  not  just  the  effectiveness  of  risk  management  but  rather  the  effectiveness  of 
acquisition  management  as  a  whole.  In  light  of  the  decisions  leading  to  the  current  state 
of  the  F-35,  it  seems  to  make  little  sense  to  evaluate  the  effectiveness  of  risk  management 
policies  if  the  policies  themselves  aren’t  followed. 
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Survey  Design 

Since  a  survey  will  be  used  to  partially  answer  the  research  questions,  some 
review  of  that  literature  is  warranted  here.  However,  more  details  of  the  survey  will  be 
presented  in  the  chapter  on  methodology. 

One  consideration  that  governs  survey  use  is  determination  of  sample  size. 

Cohen  (1992)  has  provided  some  meaningful  insight  on  sample  needs  based  on  the  size 
of  the  effect  and  the  statistical  test  used.  This  article  provides  a  number  of  considerations 
for  sample  size  including  the  desired  alpha  (a)  (or  risk  of  incorrectly  rejecting  the  null 
hypothesis)  as  well  as  type  of  test,  and  effect  size. 

One  of  the  settings  for  the  level  of  significance  for  the  test,  or  alpha  (a),  is  0.05 
which  allows  for  a  5%  chance  of  falsely  rejecting  the  null  hypothesis  and  accepting  an 
effect  exists  when  it  does  not  (a  Type  I  error).  A  0.01  level  of  significance  is  sometimes 
used  but  appears  too  stringent  for  a  survey  asking  respondents  about  behavior.  The  0. 10 
level  of  significance  is  also  used  at  times  but  increases  the  chance  of  a  Type  I  error. 
Oehmen  used  a  0.01  level  of  significance  (Oehmen,  et  ah,  2014)  but,  given  a  smaller 
expected  sample  size,  using  a  0.05  level  of  significance  seems  to  make  sense  here.  This 
difference  in  significance  presents  some  challenge  but  based  on  the  expected  sample  size 
discussed  below  would  be  unavoidable.  Since  no  attempt  will  be  made  to  show  one 
sample  as  being  more  or  less  significant,  this  difference  should  be  a  minor  one. 

The  other  two  considerations  are  the  type  of  test  and  effect  size.  Commonly, 
measures  of  correlation  are  used  to  show  how  two  variables  move  together.  Regression 
models  can  be  used  to  determine  the  size  of  the  influence  in  addition  to  correlation. 
Primarily,  this  research  will  look  for  correlations  and,  if  warranted,  characterize  the 
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correlations  via  a  regression  model.  Oehmen  reported  Goodman-Kruskal  Gamma  values 
(correlation)  of  between  0.40  and  0.51.  Cohen  would  consider  this  to  be  a  medium  to 
large  effect  for  correlation  and  large  for  regression  (Cohen,  1992).  Cohen  describes  a 
medium  effect  size  as  one  that  would  “represent  an  effect  likely  to  be  visible  to  the  naked 
eye  of  a  careful  observer”  (Cohen,  1992:156). 

Based  on  the  tests  of  correlation  and  a  medium  to  large  effect  size,  Cohen 
suggests  a  sample  size  of  twenty-eight  for  a  large  effect  and  85  for  a  medium  effect  given 
a  5%  level  of  significance  (Cohen,  1992:158).  Since  a  regression  model  will  also  be 
used,  Cohen  suggests  a  sample  size  of  42  for  a  large  effect  and  a  sample  of  91  for  a 
medium  effect  (ibid). 

Summary 

The  chapter  has  reviewed  a  number  of  factors  useful  for  the  remainder  of  the 
study.  Definitions  of  “risk”  and  “risk  management”  were  explored  to  provide  a  common 
set  of  terms.  Additionally,  five  different  risk  management  frameworks  were  explored  to 
determine  what  prescriptions  exist  for  practicing  risk  management.  Additionally,  a 
review  of  different  studies  were  reviewed  to  determine  if  risk  management  might  be 
expected  to  relate  to  project  performance  relative  to  cost,  schedule,  or  performance 
success.  Those  studies  also  indicated  that  the  practices  provided  are  also  effective  in 
aiding  projects  to  meet  their  cost,  schedule,  and  technical  performance  success  targets. 
Finally,  a  literature  relevant  to  survey  research  was  provided  to  provide  a  basis  for 
reviewing  the  methodology  in  use  for  this  study. 
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III.  Methodology 


Chapter  Overview 

This  chapter  will  detail  the  research  methodology  for  this  effort.  This  research 
study’s  three  researeh  questions  will  be  answered  through  the  eolleetion  and  analysis  of 
survey  data.  Details  will  be  provided  eovering  the  population  of  interest  as  well  as  the 
expeeted  respondents.  Additionally,  information  on  the  survey  design,  eolleetion  efforts, 
sample  size,  and  analysis  will  be  diseussed. 

Evaluating  the  effectiveness  of  risk  management  requires  an  analysis  of  the 
projeet  team’s  behavior  with  respect  to  the  type  and  extent  of  risk  management  practices 
that  were  used.  While  objective,  quantitative  data  ean  be  collected  on  some  faeets  of 
projeet  performanee,  this  alone  is  insuffieient  beeause  it  would  fail  to  capture  the  aetual 
steps  taken  to  reduce  risk.  With  that  in  mind,  a  survey  will  be  used  to  evaluate  the  risk 
praetiees  that  are  employed  by  projeet  teams.  This  survey  work  should  provide  more 
insight  into  the  inner  workings  of  the  projeet  than  other  purely  objective  methods  such  as 
merely  reviewing  eost  and  schedule  performanee.  Those  other  methods  would  fail  to 
capture  the  actual  practices  in  use  in  the  project  office. 

Since  the  goals  of  a  project  are  to  meet  cost,  schedule  and  technical  performance 
targets,  these  will  be  the  dependent  variables  for  this  study.  Additionally,  a  measure  of 
customer  satisfaction  will  be  used  as  another  measure  of  project  success.  The 
independent  variables  will  be  individual  risk  praetiees  that  the  reviewed  frameworks 
prescribe  to  improve  the  likelihood  of  project  success.  These  variables  will  be  will  be 
implemented  by  a  survey  instrument  asking  respondents  about  the  practices  in  use  on  a 
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project  of  their  choosing  and  the  success  of  that  project  as  measured  by  achievement  of 
success  for  the  dependent  variables. 

Information  provided  from  the  survey  will  also  be  used  to  record  the  performance 
of  the  dependent  variables.  While  potentially  biased,  this  information  can  be  better 
matched  to  the  risk  management  practice  information  reported  by  the  respondents.  Since 
many  defense  projects/programs  extend  over  many  years,  these  respondents  have  more 
insight  into  the  project  during  the  same  specific  window  in  time  during  which  the 
practices  were  in  use.  Longer  term  studies  might  be  better  for  matching  ultimate  project 
performance  and  risk  management  practices  but  the  necessity  of  baseline  changes  during 
a  program  that  stretches  years  might  be  even  more  difficult  to  employ  accurately  than  this 
methodology. 

Population  of  Interest 

The  population  of  interest  for  this  effort  is  U.S.  Air  Force  (USAF)  acquisition 
personnel.  However,  challenges  exist  to  surveying  this  population.  For  example,  Air 
Force  policies  strictly  prohibit  the  use  of  surveys  except  in  limited  circumstances. 

Instead,  a  reasonable  subset  of  that  population  will  become  the  sample  population  of 
interest.  The  sample  population  of  interest  will  be  those  who  have  completed  AFIT’s 
Intermediate  Project  Management  Skills  Course  (IPM)  301  course.  All  of  the  personnel 
who  might  complete  the  survey  will  be  from  the  Program  Management  or  Engineering 
career  fields  since  they  represent  those  in  the  best  position  to  consider  cost,  schedule,  and 
technical  success  as  well  as  the  actual  risk  management  practices  in  use  on  that  project. 
However,  other  career  fields  such  as  Logistics,  Cost  Estimating,  Financial  Management 
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and  others  might  be  interested  in  answering  the  survey.  Any  of  these  respondents 
comprise  the  sample  under  study  and  have  similar  backgrounds  and  expertise  with 
respect  to  risk  management  practices  on  their  assigned  projects. 

Survey  Collection 

An  anonymous  survey  will  be  administered  over  a  period  of  several  months  to 
collect  data.  The  survey  will  be  provided  electronically  via  the  Air  Force  Institute  of 
Technology’s  (AFIT’s)  WebSIRS  platform  based  on  a  design  from  the  previous  work  of 
Dr.  Josef  Oehmen.  That  survey  was  previously  used  (Oehmen,  et  ah,  2014)  and  forms 
the  basis  of  this  study. 

The  sample  for  this  survey  will  be  students  attending  classes  at  AFIT.  The  intent 
of  this  survey  collection  is  to  limit  survey  responses  to  those  within  the  intended 
population  (USAF  acquisition  personnel).  Students  will  be  solicited  to  complete  the 
survey  during  time  provided  by  their  training  event.  Potential  respondents  will  be 
advised  of  the  voluntary  nature  of  their  involvement  and  other  human  subject  rights  such 
as  the  ability  to  exit  the  survey  at  any  time  without  penalty.  Given  the  limitations  of 
performing  survey  work  within  the  Air  Force,  no  attempt  is  made  to  broadcast  the 
availability  of  the  survey  by  any  means  other  than  these  class  settings. 

The  primary  means  of  collecting  survey  data,  IPM  301,  provides  access  what 
should  be  a  representative  sample  of  the  target  population.  This  course,  offered  by 
AFIT’s  School  of  Systems  and  Logistics  (AFIT/LS),  is  an  Air  Force  fulfillment  of  the 
requirement  for  two  of  the  Defense  Acquisition  University  (DAU)  Program  Management 
Tools  (PMT)  courses,  PMT  25 1  and  PMT  257.  Those  courses  focus  on  program 
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management  tools  and  are  required  for  the  DoD  Acquisition  Professional  Development 
Program  (APDP)  Level  II  Program  Management  Certification.  Because  of  that 
fulfillment,  the  course  is  in  high  demand  and  is  expected  to  draw  a  large  number  of 
students  from  the  desired  population. 

Participants  in  this  class  typically  range  in  rank  from  L'  Lieutenant  (0-2)  to 
Major  (0-4)  for  the  active  duty  students,  although  more  senior  members  do  attend 
periodically.  Civilian  participants  of  the  course  are  typically  between  GS-09  and  GS-13, 
although,  more  senior  civilian  members  may  attend  on  occasion  as  well.  The  active  duty 
students  typically  have  between  two  and  twelve  years  of  government  acquisition 
experience.  The  civilians  present  a  much  wider  range  of  experience  where  some  are 
retired  military  personnel  and  others  joined  the  civil  service  after  some  time  in  civilian 
industry.  In  short,  this  class  provides  a  wide  range  of  background  and  experience  to 
inform  the  students’  responses  to  the  survey. 

Achieving  the  desired  sample  size  is  critical  for  survey  research.  Based  on  this 
guidance  from  Cohen,  a  minimum  desired  sample  size  is  42  (in  the  event  a  large  effect  is 
expected)  and  a  sample  size  of  91  (in  the  event  a  small  effect  is  expected)  (Cohen,  1992). 
The  Oehmen  study  noted  in  their  effect  what  Cohen  might  consider  a  large  effect.  The 
actual  sample  size  needed  will  be  dictated  by  the  effect  noted  in  this  study  but  a  sample 
size  between  42  and  91  should  be  expected  to  meet  the  needs  of  the  analysis  planned. 

Survey  Details 

The  survey  for  this  study  will  be  kept  as  similar  to  the  Oehmen  study  in  all  ways 
possible.  Since  that  survey  was  designed  for  use  in  industry,  the  survey  for  this  research 
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was  adapted  to  make  it  more  meaningful  for  a  government  audience.  The  original  survey 
questions  are  provided  as  Appendix  A.  In  order  to  aid  in  comparing  results,  every  effort 
was  made  to  keep  the  questions  and  response  options  as  consistent  as  possible.  In  most 
cases,  the  questions  remained  identical  in  wording  and  answer  type.  However,  some 
questions  simply  were  not  applicable  to  the  government  so  changes  were  made.  For 
example,  the  original  industry-focused  survey  asked  about  the  use  of  cost  reimbursable 
contracts  which,  from  industry’s  perspective,  serves  to  place  risk  on  the  government. 

This  survey  instead  used  fixed  price  contracts  in  lieu  of  cost  reimbursable  contracts  for 
this  question  since,  in  this  case,  their  use  would  put  that  same  risk  on  to  the  private 
contractor.  Finally,  two  demographic  questions  were  added.  One  asked  the  respondent  if 
they  were  a  military  member,  civilian  or  contractor  and  the  other  asked  how  many  years 
of  acquisition  experience  the  respondent  has.  Other  than  these  changes,  the  survey  for 
this  study  and  the  Oehmen  study’s  survey  will  be  the  same  in  questions,  pagination  and 
grouping,  or  the  “binning”,  of  constructs. 

The  survey  is  comprised  of  four  question  categories.  First,  respondents  are  asked 
for  basic  demographic  data  concerning  professional  experience  and  career  field 
background.  Additionally,  they  are  instructed  to  answer  all  of  the  questions  using  the 
same  project  as  a  reference  point.  Second,  respondents  are  asked  about  a  variety  of 
organizational  factors.  These  organizational  factors  include  budget,  development 
approaches/methodologies  and  specifically  which  risk  management  models  are  in  use. 
This  information  will  potentially  be  used  for  exploratory  analysis  to  look  for  other  factors 
related  to  risk  management  performance.  The  third  set  of  questions  deal  specifically  with 
risk  management  processes.  Risk  management  process  areas  include  planning,  risks  and 
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risk  analysis,  risk  mitigation,  and  risk  monitoring.  Finally,  in  the  fourth  set  of  questions, 
respondents  are  asked  to  provide  information  on  risk  management  performance  where 
they  have  the  opportunity  to  provide  insight  on  how  effective  efforts  are  at  improving 
project  or  program  success. 

In  total,  the  survey  in  use  for  this  study  comprises  173  questions  and  it’s  expected 
to  take  approximately  30  minutes  to  complete.  This  includes  ten  demographic  questions 
and  twenty  additional  questions  pertaining  to  a  project  in  which  the  subject  was  involved. 
Within  the  173  questions,  are  an  additional  seventeen  questions  on  risk  management 
planning.  Most  of  the  questions  to  this  point  in  the  survey  are  on  a  five-point  Likert  scale 
(i.e.,  “strongly  disagree”  to  “strongly  agree”).  There  are  a  few  open-ended  questions  to 
allow  for  free  entry  of  text  as  well  as  three  questions  which  allow  for  the  selection  of 
more  than  one  response. 

Table  6,  below,  provides  a  summary  of  the  questions  on  the  survey. 

A  few  of  the  survey  questions  will  not  be  considered  for  this  study.  Questions  1 
through  30  had  more  to  do  with  demographics  of  the  respondent.  These  will  be  reviewed 
for  demographic  information  but  will  not  be  part  of  the  statistical  analysis.  Additionally, 
Questions  48  through  77  asked  the  respondents  about  the  impact  of  risks  that  had 
happened.  The  binary  and  free  text  questions  were  included  as  qualitative  questions  for 
exploratory  reasons.  These  questions  were  posed  for  use  with  a  different  methodology. 
They  are  not  useful  for  the  current  research  questions  and  methodology,  however,  and 
thus  will  not  be  included  in  the  statistical  analysis. 
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Table  6.  Survey  Questions  and  Answer  Types 


Question  Category 

Answer  Type 

Number  of  Questions 

Demographics  - 
Organization/Individual 

Mixed  (drop  down,  binary, 
free  text) 

10 

Demographics  - 
Project/Program 

Mixed  (drop  down,  Likert, 
binary,  free  text) 

9 

Project/Program  -  Challenges 
Experienced 

Five-point  Likert  (“very 
low”  to  “very  high”)  with 
one  free  text 

11 

Risk  Management  -  Planning 

Five-point  Likert  (“strongly 
disagree”  to  “strongly 
agree”)  with  one  binary  and 
one  free  text 

17 

Risk  Management  -  Risks  and 
Impacts 

Six-point  Likert  (“not 
occurred”  to  “  very  high 
impact”)  with  five  free  text 

30 

Risk  Management  -  Analysis 
and  Quantification 

Five-point  Likert  (“never”  to 
“always  used”)  with  one 
binary  and  one  free  text 

7 

Risk  Management  -  Risk 
Evaluation 

Five-point  Likert  (“never”  to 
“always  used”)  with  one  free 
text 

9 

Risk  Management  -  Risk 
Mitigation 

Six-point  Likert  (“not  used” 
to  “very  high  risk  reduction 
achieved”)  with  one  binary 
and  five  free  text 
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Risk  Management  - 
Monitoring  and  Review 

Five-point  Likert 
(“completely  disagree”  to 
“completely  agree”) 

5 

Risk  Management  -  Formal 
Review 

Nine-point  Likert  (“daily”  to 
“only  after  specific  events”) 
with  two  binary  and  one  free 
text 

9 

Risk  Management  - 
Performance 

Five-point  Likert  (“strongly 
disagree”  to  “strongly 
agree”) 

22 

Overall  Project  Success 

Five-point  Likert  (“complete 
failure  to  meet  target  by 
more  than  30%”  to  “Strongly 
exceeded  target  by  30%”) 

6 
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Questions  153,  155,  162,  and  166  asked  the  respondent  about  the  outcome  of  risk 
management  (i.e.,  organization  is  satisfied  with  RM  performance,  the  ROI  of  RM  is 
positive,  etc.).  Since  this  study  is  using  the  cost,  schedule,  technical  performance,  and 
customer  satisfaction  success  variables  as  outcome  variables,  these  questions  did  not  add 
any  additional  insight  and  were  ignored.  These  were  kept  in  the  original  survey  to  match 
the  original  survey  as  much  as  possible.  However,  the  recently  published  Oehmen  article 
did  not  center  on  these  questions  so  they  will  not  be  used  for  this  study  either.  This 
reduced  the  number  of  useful  questions  for  correlation  and  statistical  analysis  from  173  to 
95  for  this  study. 

The  dependent  variables  for  this  survey  map  to  questions  168-171  and 
correspond  to,  respectively,  cost  success,  schedule  success,  technical  performance 
success,  and  customer  satisfaction.  All  four  questions  are  answerable  with  a  five-point 
Likert  scale  ranging  from  “complete  failure  to  meet  target  (by  more  than  30%)”  to 
“Strongly  exceeded  our  target  (by  more  than  30%).”  The  answers  for  cost  and  schedule 
may  be  fairly  simple.  Both  can  be  measured  on  a  numeric  scale  of  some  kind— cost  in 
currency  and  time  in  days,  weeks,  etc.  The  technical  performance  measure  is  a  little 
more  difficult.  Few  large  systems  have  a  single  dimension  of  technical  performance  so 
the  respondent  may  be  forced  to  make  broad  generalization  from  their  view  of  the  project 
which  could  be  limited  to  their  technical  area  of  expertise. 

The  measure  for  customer  satisfaction  presents  some  challenge.  For  one,  since 
most  of  the  frameworks  above  agree  that  the  three  goals  of  a  project  are  cost,  schedule, 
and  scope/performance,  customer  satisfaction  would  presumably  be  a  function  of  the 
other  three  variables.  Additionally,  the  satisfaction  score  is  reported  by  someone  in  the 
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developing  organization,  not  someone  from  the  client’s  or  customer’s  organization. 

While  interesting  as  an  outcome  variable,  this  one  presents  the  most  challenge  for 
drawing  inferences. 

However,  the  information  collected  from  this  variable  could  still  represent  useful 
information.  Notionally,  a  customer’s  satisfaction  would  be  a  function  of  cost,  schedule 
and  performance.  However,  military  users  might  be  different.  Since  the  user  of  the 
delivered  service  or  product  is  likely  not  paying  directly  for  the  product,  they  may  be 
disinterested  in  the  cost  of  item  unless  is  directly  and  immediately  influences  the  quantity 
available  for  purchase.  Even  then,  it  might  not  be  immediately  known.  While  the  user 
might  like  the  deliverable  sooner,  some  have  argued  that  the  military  user  is  most 
interested  in  technical  performance  and  most  flexible  on  cost  (e.g.,  Wirthlin,  2009).  With 
these  thoughts  in  mind,  this  dependent  variable  will  still  be  explored  since  it  provides  a 
useful  measure  different  from  the  other  three  dependent  variables. 

Survey  Data  Analysis 

The  survey  data  will  be  analysis  through  a  variety  of  steps.  Once  the  surveys  are 
collected,  the  raw  data  will  be  processed  to  remove  blank  or  incomplete  entries.  Since 
questions  may  be  skipped,  some  missing  data  may  exist  but  will  be  left  in  the  data  set 
provided  the  respondent  made  it  far  enough  into  the  survey  to  begin  answering  questions 
on  risk  practices.  All  of  the  data  statistical  analysis  will  be  performed  using  SPSS. 

Prior  to  survey  analysis,  a  few  questions  will  be  adjusted.  The  response  scale  for 
questions  137  through  140  will  be  reverse  coded.  Originally,  these  response  options  were 
ranked  high  to  low  where  most  of  the  other  Likert  options  in  the  survey  ranged  low  to 
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high.  Additionally,  Questions  156  through  160  will  be  reverse  coded  for  the  same 
reason.  Initially,  the  questions  ranked  most  to  least  frequent  with  respect  to  reviewing  the 
risk  management  effort.  After  these  changes,  all  of  the  survey  responses  can  be  matched 
against  each  other  in  a  consistent  manner  on  the  same  ordinal  scale. 

The  research  questions  and  their  methodologies  are  summarized  in  Table  7  below. 


Table  7.  Research  Questions  and  Methodologies 


Question 

Hypothesis 

Methodology 

Ql.  Does  a  relationship  exist 
between  risk  management  and 
an  Air  Force  project  meeting 
its  cost,  schedule,  and 
performance  goals? 

HI.  There  exists  a 
statistically  significant 
correlation  between  risk 
management  practices  and 

Air  Force  project  success 
relative  to  cost,  schedule  or 
performance  goals. 

Survey  correlation 
analysis 

Q2.  How  effective  is  the  Air 
Force  risk  management  effort 
in  aiding  its  projects  to  meet 
their  given  cost,  schedule  and 
performance  targets? 

H2.  At  least  one  risk 
management  practice  will  be 
shown  to  significantly  affect 
Air  Force  project  success 
relative  to  cost,  schedule  or 
performance  goals. 

Survey  correlation 
analysis  combined  with 
factor/regression 
analysis 

Q3.  Are  there  differences 
between  the  effective  risk 
management  practices  used  in 
industry  and  those  used  by  the 
Air  Force? 

H3.  A  difference  does  not 
exist  between  the  effective 
industry  practices  shown  to 
relate  to  affect  Air  Force 
project  success  relative  to 
cost,  schedule  or  performance 
goals. 

Survey  correlation 
analysis  comparing 
these  results  with  the 
Oehmen  study  results 

Table  7  above  will  be  helpful  for  following  the  structure  of  the  methodologies 
described  in  this  chapter  as  well  as  the  analysis  performed  in  the  subsequent  chapter. 
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Research  Question  1  Methodology 

This  research  question  will  be  answered  by  performing  a  correlation  analysis 
comparing  individual  questions  with  the  four  dependent  variables.  To  ease  both 
understanding  and  clarity  in  answering  the  research  questions,  the  individual  survey 
questions  will  be  analyzed  here  in  bins  that  match  the  risk  framework  steps  for  this  study. 
This  framework  will  make  clear  any  relationship  risk  management  has  with  project 
success  outcomes.  This  correlation  analysis  will  be  performed  using  Pearson’s 
correlation  coefficient  and  examine  if  individual  questions  have  a  relationship  with  the 
four  dependent  outcome  variables.  This  question  can  be  answered  by  looking  for 
statistically  significant  correlations  between  the  individual  questions  and  the  outcome 
variables. 

The  null  hypothesis  for  this  question  is  that  no  relationship  exists  between  risk 
management  and  project  success  with  respect  to  cost,  schedule,  technical  performance, 
and  customer  satisfaction.  If  the  null  hypothesis  cannot  be  rejected,  the  study  will 
conclude  that  no  relationship  exists  for  this  sample  population. 

Research  Question  2  Methodology 

The  previous  correlation  analysis  will  also  be  used  to  partly  answer  the  second 
research  question.  Since  this  question  asks  about  the  effectiveness  of  risk  management, 
the  magnitude  and  direction  of  the  correlation  will  also  be  reviewed.  Finally,  that 
correlation  analysis  will  be  used  as  the  basis  for  answering  the  third  research  question 
through  the  use  of  a  factor  analysis  followed  by  a  multiple  linear  ordinary  least  squares 
regression. 
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This  study  will  attempt  to  preserve  the  same  basie  analytical  model  as  the 
Oehmen  study  and  make  use  of  the  same  factors.  If  possible,  this  study  will  attempt  to 
construct  and  test  the  same  factors  used  in  the  Oehmen  study.  These  factors  are  the  same 
as  those  provided  in  Table  4  where  they  were  mapped  to  the  individual  risk  framework 
steps.  Each  dependent  factor  variable  will  be  tested  for  its  relationship  to  all  six 
independent  variables.  It  is  hypothesized  that  all  six  independent  variables  are  positively 
related  to  the  dependent  variables.  The  planned  independent  and  dependent  variables  are 
displayed  in  Table  8  below. 


Table  8.  Oehmen  Survey’s  Independent  and  Dependent  Variables 


Independent  Factors 

Dependent  Variables 

IVl. 

Risk  Management  Skills  and 

DVl. 

Project  Cost  Success 

Resources 

DV2. 

Project  Schedule  Success 

IV2. 

Risk  Management  Tailoring 

DV3. 

Product  Technical  Success 

IV3. 

Risk  Impact  Quantification 

DV4. 

Customer  Satisfaction 

IV4. 

Risk  Based  Decision  Making 

IV5. 

Risk  Monitoring  and  Review 

IV6. 

Transparency 

(Oehmen,  et  ah,  2014) 

These  independent  factors  describe  a  number  of  constructs.  The  first  independent 
factor  described  above  describes  how  well  skills  and  resources  are  provided  to  deal  with 
risk  management.  Since  all  projects  are  unique,  the  second  factor  describes  how  well  the 
risk  effort  was  tailored  to  deal  with  the  specific  project  under  analysis.  The  third  factor, 
risk  quantification,  describes  how  well  the  identified  risks  were  quantified  as  part  of  the 
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analysis  step.  The  fourth  factor  describes  how  well  the  risk  analysis  was  included  in 
decision  making  for  the  project  and  organization.  The  fifth  factor,  risk  monitoring  and 
review,  focuses  on  how  well  risks  were  tracked  during  project  execution.  Finally, 
transparency  measures  how  well  risks  were  communicated  though-out  the  organization(s) 
involved  in  the  project. 

In  the  event  the  constructs  cannot  be  recreated,  questions  with  a  statistically 
significant  correlation  will  be  used  in  an  exploratory  factor  analysis  to  construct  factors 
that  can  be  used  in  a  multiple  linear  regression  model.  The  exploratory  factor  analysis 
will  be  performed  to  examine  the  dimensionality  of  the  constructs  involved.  This  factor 
analysis  will  seek  to  identify  the  underlying  relationships  between  measured  variables 
and  provide  support  for  the  latent  constructs  in  the  survey. 

Reliability  analysis  will  be  performed  on  the  individual  questions  using 
Chronbach’s  Alpha  as  the  test  statistic  (George  and  Mallery,  2003;  Kline,  1999).  An 
Alpha  of  0.700  or  greater  will  be  used  as  support  to  aggregate  the  individual  questions 
into  an  overall  latent  variable  for  each  of  the  six  independent  variables  listed  above. 
Aggregated  factors  will  be  calculated  by  averaging  the  score  of  the  individual  questions 
within  those  factors/variables. 

A  review  of  the  descriptive  statistics  will  be  performed  on  the  factors.  This  will 
assess  normality  of  the  data.  Additional  statistical  analysis  may  be  required  in  the  event 
the  data  fails  this  test  of  normality.  Aggregated  factor  means,  standard  deviations  as  well 
as  kurtosis  and  skewness  will  be  reviewed  to  ensure  the  measures  are  sound  for  additional 
analysis. 
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Assuming  normality  with  the  individual  factor  averages,  a  correlation  table  using 
Pearson’s  correlation  coefficient  will  be  examined.  This  correlation  table  will  be  used  to 
assess  the  relationship  the  constructed  factors  have  with  the  four  dependent  variables. 

This  relationship  will  be  reviewed  for  both  significance  as  well  as  the  strength  and 
direction  of  the  relationship  (positive  or  negative). 

Finally,  four  multiple  linear  regressions  will  be  performed  to  model  the 
relationship  between  the  independent  factors  and  each  of  the  four  dependent  variables 
(cost,  schedule,  technical  performance,  and  customer  satisfaction).  The  models  will  be 
examined  to  determine  what  significance  and  relationship  the  individual  factors  have 
with  each  of  the  four  dependent  variables. 

The  null  hypothesis  for  this  question  is  that  risk  management  is  not  effective  in 
aiding  the  project  in  meeting  its  success  target  with  respect  to  cost,  schedule,  technical 
performance,  and  customer  satisfaction.  If  the  null  hypothesis  cannot  be  rejected,  the 
study  will  conclude  that  risk  management  is  not  effective. 

Research  Question  3  Methodology 

Since  the  Oehmen  study  did  not  perform  any  regression  analysis,  any  comparison 
to  that  study  will  be  limited  to  comparing  correlation  tables  only.  This  research  question 
will  be  answered  by  using  the  correlation  results  used  to  answer  the  first  two  research 
questions  and  comparing  those  to  the  results  published  for  the  Oehmen  study.  Significant 
questions  from  the  two  studies  will  be  compared  to  determine  if  they  are  the  same 
questions  and  to  evaluate  the  direction  of  the  relationship  (positive  or  negative).  This 
study  uses  Pearson’s  correlation  coefficient  where  the  Oehmen  study  uses  Kruskal- 
Goodman’s  Gamma.  Since  they  are  different  measures  of  correlation,  no  determination 
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can  be  made  about  a  given  question  being  more  or  less  correlated  but  the  two  can  still  be 
compared  for  significanee  and  direction  of  impact. 

The  null  hypothesis  for  this  question  is  that  a  difference  does  exist  between  the 
effeetive  risk  management  praetiees  used  in  industry  and  those  used  in  the  Air  Foree.  If 
the  null  hypothesis  cannot  be  rejected,  the  study  will  conclude  that  a  differenee  does  exist 
between  the  industry  practices  and  the  Air  Force  praetiees.. 

Summary 

This  ehapter  details  the  method  for  answering  the  three  research  questions.  A 
survey  will  be  administered  to  Air  Foree  aequisition  personnel  who  represent  the  sample 
population  of  interest.  Data  from  this  survey  instrument  will  be  used  to  perform  a 
correlation  analysis  to  answer  the  first  two  researeh  questions.  Finally,  a  factor  analysis 
and  regression  analysis  will  be  performed  to  answer  the  third  research  question. 
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IV.  Analysis  and  Results 


Chapter  Overview 

A  number  of  analysis  steps  are  performed  to  answer  the  research  questions.  First, 
some  preliminary  survey  data  analysis  is  performed  to  evaluate  the  demographics  of  the 
respondents.  A  correlation  analysis  is  performed  to  answer  the  first  research  question. 
Those  results  are  used  along  with  a  factor  analysis  to  perform  regression  models  for  each 
of  the  four  dependent  variables  to  answer  the  second  research  question.  Finally,  results 
from  this  survey  are  compared  to  the  survey  performed  by  Oehmen,  et  al.  to  determine 
what,  if  any,  difference  exists  between  those  two  samples. 

Survey  Collection 

Survey  responses  were  collected  from  6  June  2013  to  12  March  2015.  Surveys 
were  expected  to  come  from  three  different  training  class  sources  but,  in  the  end,  all  or 
nearly  all  of  the  responses  came  from  AFIT’s  IPM  301  course.  Historically,  the  course  is 
attended  almost  exclusively  by  acquisition  program  managers  with  the  exception  of  the 
occasional  engineer  or  logistician.  The  actual  student  demographics  for  this  course 
match  those  expectations  detailed  in  Chapter  2,  but  the  total  number  of  responses  was 
less  than  expected.  In  total,  190  students  started  the  survey,  with  only  127  of  them 
completing  enough  of  the  survey  for  it  to  be  of  analytical  value  in  this  study.  Depending 
on  the  variable  in  question,  the  number  of  useful  responses  was  less  than  80.  Based  on 
Cohen’s  recommendations  regarding,  this  is  well  above  the  lower  number  required,  and 
close  to  the  higher  number,  so  this  should  be  sufficient  depending  on  the  strength  of 
impact  found  from  the  data  analysis. 
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In  addition  to  the  challenges  encountered  in  collecting  surveys  from  the  two 
partner  organizations  (DAU  and  AFLCMC),  even  the  AFIT  students  had  difficulty 
completing  the  survey.  The  survey  was  hosted  on  AFIT’s  WEBSirs  electronic  survey 
platform.  On  numerous  occasions,  students  experienced  system  outages  when  attempting 
to  access  the  survey.  In  some  cases,  the  electronic  survey  was  restored  in  time  for  those 
students  to  complete  the  survey  prior  to  the  end  of  their  course.  However,  in  a  few  of 
cases,  students  completed  their  two  week  course  without  being  able  to  access  the  survey. 
These  problems  also  contributed  to  a  lower  number  of  survey  responses  than  expected. 

Respondents  were  asked  a  number  of  demographic  questions.  One  of  those 
questions  asked  about  their  amount  of  acquisition  experience.  This  is  summarized  in 
Table  9  and  Figure  1  below. 


Table  9.  Survey  Respondent's  Years  of  Acquisition  Experience 


Frequency 

Percent 

Valid  Percent 

Cumulative 

Percent 

Missing 

1 

.8 

.8 

.8 

0-5  years 

80 

64.5 

64.5 

65.3 

6-10  years 

21 

16.9 

16.9 

82.2 

11-15  years 

7 

5.6 

5.6 

87.8 

16-20  years 

6 

4.8 

4.8 

92.6 

20+  years 

9 

7.3 

7.3 

100.0 

Total 

124 

100.0 

100.0 

50 


■  Missing 

■  0-5  years 

■  6-10  years 

■  11-15  years 

■  16-20  years 

■  20+  years 


Figure  1.  Survey  Respondent's  Years  of  Acquisition  Experience 


Since  the  course  is  targeted  at  students  with  three  years  of  acquisition  experience, 
the  larger  number  of  less  experienced  respondents  is  expected.  However,  as  shown  in  the 
table  above,  the  course  also  draws  more  experienced  personnel.  This  is  likely  due  to  the 
course’s  status  as  a  fulfillment  of  certification  requirements  for  their  job  positions. 

Respondents  were  also  asked  about  their  employment  status  at  the  time  they  filled 
out  the  survey.  These  results  are  summarized  in  Table  10  and  Figure  2  below. 


Table  10.  Survey  Respondent's  Employment  Status 


Frequency 

Percent 

Valid  Percent 

Cumulative 

Percent 

Active  Duty 

63 

50.8 

50.8 

50.8 

Civilian 

60 

48.4 

48.4 

99.2 

Contractor 

1 

.8 

.8 

100.0 

Total 

124 

100.0 

100.0 

51 


Figure  2.  Survey  Respondent's  Employment  Status 


One  respondent  identified  him  or  herself  as  being  a  contractor  working  directly  in 
a  government  program  office.  While  contractors  may  have  work  that  is  different  from 
direct  Air  Force  employees,  a  contractor  working  in  a  direct  support  position  in  the 
government  program  office  might  be  expected  to  have  similar  work  history  and  views  as 
the  direct  employees  and  so  these  responses  were  deemed  to  be  valid  for  inclusion  in  this 
study. 

Survey  Data  Analysis 

To  match  the  original  Oehmen  study,  this  sample  was  reviewed  for  correlations 
comparing  the  relationship  between  the  independent  questions  and  the  final  four 
dependent  questions  on  outcome  success  (cost,  schedule,  technical  performance,  and 
customer  satisfaction).  Using  Pearson’s  correlation  coefficient  (r),  a  number  of  variables 
were  found  to  be  statistically  significant  in  their  covariance  with  the  four  outcome 
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(dependent)  variables.  Some  of  these  questions  mateh  those  diseovered  by  the  Oehmen 
artiele,  while  some  did  not.  Additionally,  an  exploratory  factor  analysis  is  presented  and 
multiple  regression  models  show  the  relationships  those  factors  have  with  the  dependent 
variables. 

The  methodology  for  answering  the  three  research  questions  will  now  be 
discussed. 

Research  Question  1:  Correlation  Analysis 

The  first  research  question  asks  if  a  relationship  exists  between  risk  management 
and  the  Air  Force  project  meeting  its  cost,  schedule,  and  performance  success  targets. 
The  following  section  will  analyze  the  survey  data  in  an  effort  to  answer  that  question  by 
examining  the  correlations  of  each  risk  management  step. 

This  study  found  a  much  larger  list  of  variables  to  be  significantly  correlated  with 
the  four  outcome  variables  than  what  Oehmen  found.  However,  this  may  be  due  to  the 
difference  in  the  selected  level  of  significance.  Recall  that  Oehmen  selected  the  0.01 
level  of  significance  whereas  this  study  uses  the  0.05  level  of  significance. 

This  study  finds  a  total  of  48  of  the  109  questions  in  use  to  be  significantly 
correlated  (as  measured  by  the  Pearson  correlation  coefficient)  with  the  four  selected 
outcome  variables.  Thirty  of  these  questions  are  related  to  risk  management  practices 
associated  with  planning  and  preparation  to  reduce  risk.  The  additional  eighteen 
questions  were  found  to  be  significant  correlations  between  specific  risk  handling  plans 
and  are  described  later.  All  of  the  significant  variables  had  sample  sizes  between  74  and 
82. 
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The  statistically  significant  correlations  are  provided  in  a  series  of  tables  below 
(along  with  their  p-values  in  parentheses)  based  on  the  risk  management  frameworks 
discussed  in  Chap  2.  This  will  improve  clarity  as  well  as  provide  additional  support  for 
the  frameworks  themselves.  Note  that  some  of  the  names  of  the  dependent  variables  in 
the  columns  have  been  shortened  from  schedule  success,  technical  performance  success, 
and  customer  satisfaction  success  to  “Sched.  Success,”  “Tech.  Perf  Success,”  and  “Cust. 
Sat.  Success,”  respectively. 

Risk  Management  Planning 

Recall  that  the  first  risk  management  step  for  use  in  this  study  is  risk  management 
planning.  These  actions  are  taken  by  the  organization  to  focus  the  risk  management  on 
the  specific  project  at  hand.  The  correlation  results  for  the  significant  risk  management 
planning  questions  are  provided  in  Table  1 1  below.  The  top  number  in  each  data  cell  is 
the  correlation  coefficient  with  the  corresponding  p-value  shown  in  parentheses  below. 


Table  11.  Significantly  Correlated  Planning  Variables 


Risk  Planning  Variables 

Cost 

Success 

Sched. 

Success 

Tech.  Perf. 
Success 

Cust.  Sat. 
Success 

Q3 1 .  Our  employees  are  motivated  to 
perform/implement  risk  management 

- 

- 

- 

0.295 

(0.008) 

Q35.  RM  is  systematic,  structured 
and  timely 

- 

- 

- 

0.238 

(0.032) 

Q45.  RM  teams  are  cross-functional 
and  cross-organizational 

- 

0.236 

(0.033) 

- 

0.268 

(0.015) 

Table  1 1  shows  that  project  performance  was  significantly  correlated  with  three 


instances  (of  the  fifteen  questions  used)  pertaining  to  the  risk  planning  phase.  Schedule 
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and  customer  satisfaction  outcomes  are  correlated  with  three  questions  but  no  questions 
were  found  to  relate  to  either  cost  or  technical  project  performance. 

Interestingly,  having  employees  motivated  to  perform  risk  management  is  found 
to  be  highly  significantly  correlated  with  customer  satisfaction  success  (p-value:  0.008). 
While  none  of  the  reviewed  frameworks  discuss  motivation  explicitly,  it  seems  to  make 
logical  sense  that  personnel  will  perform  better  with  greater  motivation. 

These  three  risk  planning  variables  provide  some  evidence  for  a  relationship 
between  risk  management  and  project  outcomes. 

Risk  Identification 

Recall  that  the  second  step  in  the  risk  management  framework  for  this  study  is 
risk  identification.  None  of  the  questions  corresponded  with  risk  identification  so  no 
results  are  presented  here. 

Risk  Analysis 

The  following  step  is  risk  analysis.  Those  questions  found  to  have  significant 
correlations  are  summarized  in  Table  12  below. 

Table  12  shows  that  project  performance  significantly  correlates  with  nine 
instances  (of  the  thirteen  questions  used)  pertaining  to  the  risk  analysis  phase.  At  least 
one  of  the  questions  correlates  with  at  least  one  of  three  dependent  variables:  schedule, 
technical  performance,  and  customer  satisfaction  outcomes  were  all  correlated  with  three 
questions.  None  of  the  questions  correlate  with  cost  success. 
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Table  12.  Significantly  Correlated  Risk  Analysis  Questions 


Risk  Analysis  Variables 

Cost 

Success 

Sched. 

Success 

Tech.  Perf. 
Success 

Cust.  Sat. 
Success 

Q80.  Assessment  of  risks  on 
scales 

- 

- 

0.235 

(0.035) 

- 

Q83.  Monte  Carlo  simulations 
to  aggregate  different  types  of 
risk  estimates 

- 

- 

0.253 

(0.025) 

Q85.  Make  go/no  go  decisions 
based  on  risk  assessment 

- 

- 

0.284 

(0.010) 

- 

Q86.  Resources  are  allocated 
to  reduce  target  risks  as  early 
as  possible 

- 

- 

- 

0.230 

(0.040) 

Q87.  Risk  assessments  are 
used  to  set  more  ‘realistic’  or 
‘achievable’  objectives 

- 

0.231 

(0.037) 

- 

- 

Q89.  The  results  of  the  risk 
analysis  are  considered  in 
making  technical,  schedule 
and/or  cost  trade-offs 

- 

- 

0.272 

(0.013) 

- 

Q90.  Decisions  are  made  based 
on  risk-benefit  trade-offs 

- 

0.241 

(0.030) 

0.281 

(0.011) 

0.306 

(0.006) 

Q91.  Risk-benefit  trade-offs 
are  used  systematically  to  favor 
‘low-risk,  high-benefit’  options 
and  eliminate  ‘high-risk,  low- 
reward’  options 

- 

0.271 

(0.014) 

0.259 

(0.019) 

0.234 

(0.036) 

Q92.  Contracts  are  derived 
from  detailed  cost  risk 
assessments 

- 

0.230 

(0.039) 

0.237 

(0.033) 

0.251 

(0.025) 

Seven  questions  are  significantly  correlated  with  technical  performance  success. 
The  three  questions  regarding  trade-off  studies  and  contract  type  are  also  found  in  this 
group.  The  additional  questions  also  deal  with  trade-off  decision  making  with  the  final 
two  questions  dealing  with  how  the  analysis  was  performed.  The  practice  of  assessing 


56 


risks  on  scales  and  the  performance  of  simulations  were  also  significantly  and  positively 
correlated  with  technical  performance  success. 

Four  of  the  questions  related  to  Risk  Analysis  are  significantly  correlated  with 
schedule  success  and  four  are  found  to  correlate  significantly  with  customer  satisfaction. 
These  two  outcome  variables  share  three  of  the  questions  (regarding  trade-off  decisions 
and  contract  type).  All  of  the  questions  here  are  positively  correlated  and  provide  some 
non-negative  support  for  the  guidance  provided  by  the  reviewed  frameworks. 

These  twelve  risk  planning  variables  provide  some  evidence  for  a  relationship 
between  risk  management  and  project  outcomes. 

Risk  Mitigation 

A  number  of  questions  asked  respondents  about  risk  reductions  achieved  through 
the  implementation  of  specific  practices.  This  set  is  analogous  to  the  risk  mitigation  and 
implementation  step  in  use  for  this  study.  They  also  provide  some  insight  into  specific 
practices  found  to  correlate  with  the  cost,  schedule,  technical  performance,  and  customer 
satisfaction  outcomes.  As  reviewed  above,  risk  mitigation  plays  an  important  part  of  the 
risk  management  frameworks  analyzed.  The  significant  correlations  are  provided  in 
Table  13. 

Table  13  shows  that  project  performance  was  significantly  correlated  with 
eighteen  instances  (of  the  thirty-two  questions  used)  pertaining  to  the  risk  mitigation 
phase.  This  set  might  provide  the  best  support  for  risk  management  in  that  these 
questions  ask  about  the  amount  of  risk  reduction  achieved.  In  other  words,  these 
questions  provide  some  insight  into  the  types  of  risk  mitigation  pursued  to  achieve  the 
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Table  13.  Achieved  Risk  Mitigation  Actions  Significantly  Correlated  with  Outcomes 


Risk  Mitigation  Variables 

Cost 

Success 

Sched. 

Success 

Tech.  Perf. 
Success 

Cust.  Sat. 
Success 

Q96.  Contractor/Go vemment 
integration 

- 

0.284 

(0.011) 

0.344 

(0.002) 

0.323 

(0.004) 

Q97.  Organization-internal 
integration 

- 

- 

0.316 

(0.005) 

0.312 

(0.005) 

Q99.  Schedule  reserves 

- 

- 

0.261 

(0.020) 

- 

QIOO.  Contractual  sharing  of  cost 
overruns  with  contractor(s) 

- 

0.337 

(0.003) 

- 

- 

QlOl.  Contractual  sharing  of  cost 
overruns  with  other  organizations 

- 

0.292 

(0.010) 

- 

- 

Q106.  Detailed  cost,  schedule,  and 
performance  simulations  and  trade¬ 
off  studies 

- 

- 

0.327 

(0.003) 

0.316 

(0.004) 

Q108.  More  detailed  design  reviews 

- 

0.247 

(0.027) 

0.379 

(0.000) 

0.305 

(0.006) 

Q109.  Training  program  or  special 
career  path  to  increase  skill  level 

- 

0.245 

(0.032) 

- 

- 

QUO.  Define  ‘standard  work’  or 
‘standard  processes’  to  increase 
process  reliability 

- 

- 

- 

0.285 

(0.011) 

Q1 13.  Active  internal  lobbying 
toward  top  management  to  promote 
project 

- 

0.250 

(0.028) 

0.243 

(0.032) 

- 

Q1 17.  Management  of  requirements 

- 

0.264 

(0.021) 

- 

- 

Q1 18.  Active  lobbying  with  key 
stakeholders  outside  of  direct 
customer/contractor  relationship 

- 

0.320 

(0.006) 

0.291 

(0.011) 

- 

Q122.  Increased  testing  and 
prototyping  activities 

- 

- 

0.222 

(0.048) 

- 

Q124.  Develop  flexible  product 
architecture 

- 

- 

0.243 

(0.031) 

- 

Q125.  Strict  configuration  control 

- 

- 

0.348 

(0.002) 

- 

Q126.  Engineering  with  redundancy 
or  safety  margins 

- 

- 

0.290 

(0.010) 

- 

Q127.  Pursue  several  engineering 
solutions  in  parallel 

- 

- 

0.249 

(0.026) 

- 

Q128.  Focus  on  design  for 
manufacturing  or  design  for  service 

- 

- 

0.254 

(0.026) 

- 

cost,  schedule,  technical  performance,  and  customer  satisfaction  outcome  success  noted 
by  the  respondents.  Still,  these  measures  look  only  at  correlation  and  not  causation,  so  a 
clear  a  line  between  these  results  and  risk  management  effectiveness  remains  elusive. 
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Interestingly,  none  of  the  Risk  Mitigation  questions  correlate  significantly  with 
the  cost  success  outcome  variable.  While  these  variables  continue  to  provide  evidence 
that  some  positive  relationship  exists  between  a  long  list  of  example  risk  mitigation 
options,  this  lack  of  influence  on  cost  success  remains  interesting  and  unexplained 
directly  by  the  data. 

This  group  also  includes  seven  questions  that  are  highly  significantly  correlated 
with  at  least  one  of  the  dependent  variables.  These  include  the  integration  of  risk 
management  within  the  organization  as  well  as  between  the  customer  and  developer 
(govemment/contractor).  Interestingly,  contractual  sharing  of  cost  overruns  had  no 
relationship  with  cost  but  does  positively  relate  to  schedule  success.  Engineering  and 
considerations  such  as  trade-offs,  design  reviews,  and  configuration  control  were  also 
found  to  be  highly  significantly  correlated  with  technical  performance  and/or  customer 
satisfaction  success.  The  final  significantly  correlated  questions  covers  lobbying  with 
key  stakeholders  outside  of  the  customer/contractor  relationship. 

These  eighteen  risk  mitigation  variables  provide  some  evidence  for  a  relationship 
between  risk  management  and  project  outcomes. 

Risk  Tracking 

The  next  set  of  correlations  deals  with  Risk  Tracking.  This  step  follows  the 
implementation  of  the  selected  risk  responses  to  reduce  risk  likelihood  and/or  impact  and 
is  sometimes  referred  to  as  risk  monitoring  and  review.  In  total,  five  questions  are  found 
to  significantly  correlate  with  the  outcome  variables.  These  results  are  summarized  in 
Table  14  below. 
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Table  14.  Significantly  Correlated  Monitoring  and  Review  questions 


Risk  Tracking  Variables 

Cost 

Success 

Sched. 

Success 

Tech.  Perf. 
Success 

Cust.  Sat. 
Success 

Q132.  Risks  were  escalated  to  senior 

0.236 

0.279 

0.234 

management  according  to  guidelines 

(0.035) 

(0.012) 

(0.036) 

Q133.  Risks  were  regularly  re¬ 
assessed  according  to  guidelines 

- 

- 

0.315 

(0.004) 

- 

Q134.  The  RM  process  was  regularly 
reviewed  and  improved 

- 

0.248 

(0.029) 

0.251 

(0.027) 

- 

Table  14  shows  that  project  performance  was  significantly  correlated  with  three 
instances  (of  the  eleven  questions  used)  pertaining  to  the  risk  monitoring  phase.  Once 
again,  the  questions  correlate  with  schedule,  technical  performance,  and  customer 
satisfaction  but  none  for  cost  success. 

One  question,  escalating  risks  to  senior  management,  was  found  to  correlate 
significantly  and  positively  with  the  other  three  outcome  variables.  This  seems 
reasonable  in  that  senior  management  likely  has  more  resources  available  to  allocate  to 
risk  reduction.  This  would  also  seem  to  influence  costs  success  as  well.  However,  it  may 
not  appear  here  for  a  few  reasons.  For  one,  the  government  respondents  may  have 
contracted  the  developed  on  an  FFP  contract  and  would  likely  be  blind  to  the  cost  status 
of  the  project.  Alternately,  perhaps  the  risk  reduction  efforts  to  achieve  the  schedule, 
technical  performance,  and  customer  satisfaction  success  consumed  additional  financial 
resources  that,  while  it  improved  three  of  the  four  outcome  variables,  may  have  harmed 
the  cost  success  of  the  project  in  some  manner. 

Two  additional  questions  correlated  significantly  and  positively  with  technical 
performance  success.  One  of  the  questions,  how  risks  were  re-assessed,  correlates  highly 
significantly  with  only  technical  performance  success  where  the  other,  that  RM  processes 
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were  reviewed,  correlates  with  both  schedule  and  technical  performance  success.  This 
finding  is  expected  based  on  the  frameworks  reviewed  and  a  logical  interpretation  of  the 
questions. 

These  five  risk  tracking  variables  provide  some  evidence  for  a  relationship 
between  risk  management  and  project  outcomes. 

Risk  Management  Performance 

The  final  set  of  reviewed  correlations  provides  insight  to  the  organization’s  view 
of  risk  management  and  its  perceived  value.  Table  15  summarizes  these  findings  below. 

Only  three  questions  correlate  significantly  with  cost  success.  Those  three  and  an 
additional  question  correlate  significantly  with  schedule  success.  These  questions  all 
provide  some  limited  non-negative  support  for  the  effectiveness  of  risk  management. 

However,  twelve  questions  are  correlated  positively  and  significantly  with 
technical  performance  or  customer  satisfaction  success.  Ten  of  these  questions  correlate 
positively  and  significantly  with  both  outcome  variables.  In  short,  these  questions  all 
provide  non-negative  support  for  open  communication  within  the  project.  They  also 
provide  non-negative  support  for  the  notion  that  an  organization  that  perceives  value  in 
risk  management  will  also  reap  its  benefits. 

Table  15  also  shows  that  project  performance  is  significantly  correlated  with 
thirteen  instances  (of  the  twenty-two  questions  used)  pertaining  to  risk  management 
performance.  All  of  these  questions  correlate  positively  with  at  least  one  of  the  outcome 
variables  and  ten  of  them  correlate  at  the  0.01  level.  There  continue  to  be  relatively  few 
questions  that  correlate  significantly  with  the  cost  success  variable;  only  three. 
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Table  15.  Significantly  Correlated  Performance  Variables 


Risk  Management 
Performance  Variables 

Cost 

Success 

Sched. 

Success 

Tech.  Perf. 
Success 

Cust.  Sat. 
Success 

Q146.  PMs  support  RM  activities 

0.221 

(0.046) 

0.225 

(0.042) 

0.249 

(0.024) 

0.299 

(0.007) 

Q147.  RM  results  play  an  important 
part  in  the  decision  making  of  senior 
managers 

- 

0.252 

(0.022) 

0.320 

(0.003) 

0.372 

(0.001) 

QMS.  RM  results  influence  trade-off 
decisions 

- 

- 

0.266 

(0.016) 

0.243 

(0.029) 

Q150.  RM  processes  are  primary 
mechanism  to  determine  management 
reserves 

0.229 

(0.039) 

0.208 

(0.060) 

- 

- 

Q 1 5 1 .  Findings  from  the  RM 
processes  translate  into  action 

- 

- 

- 

0.293 

(0.008) 

Q154.  Budgeting  for  risks  is  an 
incentive  against  identifying  new 
risks 

- 

- 

0.264 

(0.019) 

0.335 

(0.003) 

Q156.  RM  creates  and  protects  value 

- 

- 

0.460 

(0.000) 

0.393 

(0.000) 

Q158.  RM  is  a  central  part  of 
decision  making 

- 

- 

0.349 

(0.001) 

0.399 

(0.000) 

Q159.  RM  facilitates  continuous 
improvement  in  the  organization 

- 

- 

0.301 

(0.006) 

0.458 

(0.000) 

Q160.  RM  has  a  positive  influence  on 
program  success 

- 

- 

0.352 

(0.001) 

0.448 

(0.000) 

Q161.  Project  management  took  a 
proactive  stance  in  addressing  risks 
and  issues 

- 

- 

0.337 

(0.002) 

- 

Q164.  Concerns  were  heard  and 
addressed 

0.250 

(0.023) 

0.256 

(0.020) 

0.235 

(0.033) 

0.316 

(0.004) 

Q165.  It  was  OK  to  report  ‘bad  news’ 
and  concerns 

- 

- 

0.310 

(0.005) 

0.333 

(0.002) 

Additionally,  four  of  the  variables  were  found  to  correlate  significantly  with  schedule 


success,  eleven  with  technical  performance  success,  and  eleven  were  significantly 


correlated  with  customer  satisfaction  success.  Two  of  the  questions  correlate  with  all 


four  dependent  variables. 
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These  thirteen  risk  performance  variables  provide  some  evidence  for  a 
relationship  between  risk  management  and  project  outcomes. 

Research  Question  2:  Factor  Analysis,  and  Multiple  Linear  Regression 

The  previous  correlation  analysis  used  for  Research  Question  1  will  be  used  for 
answering  this  research  question  as  well.  The  correlation  analysis  is  used  to  provide  a  list 
of  significant  questions  for  further  analysis.  That  analysis  will  be  followed  by  a  factor 
analysis  and  multiple  linear  regression  to  fully  answer  this  research  question. 

Factor  Analysis 

Since  correlation  doesn’t  provide  a  great  picture  of  relative  impact,  additional 
statistical  analysis  was  performed  to  see  which  variables  had  the  greatest  influence  on  the 
outcome  variables.  This  was  performed  with  linear  regression.  However,  additional 
question  analysis  was  required  prior  to  performing  those  regressions. 

Recall  that  the  survey  was  originally  used  by  the  Oehmen  study  to  analyze  a 
population  of  various  companies.  The  questions  were  kept  the  same  to  perform  a 
comparison  of  one-to-one  correlations  between  dependent  and  independent  variables 
among  the  two  samples  but,  for  more  detailed  statistical  analysis  for  the  Air  Force,  not  all 
questions  were  useful.  Additionally,  the  initial  factor  analysis  was  performed  using 
principal  component  analysis  as  the  extraction  method  with  varimax  orthogonal  rotation. 
Orthogonal  rotation  is  selected  when  the  variables  are  expected  to  be  uncorrelated  (Field, 
2009).  That  attempt  failed  to  find  the  same  grouping  as  used  by  Oehmen.  Next,  the 
factor  analysis  was  performed  again  using  the  oblimin  oblique  rotation  which  is  selected 
when  the  variables  are  expected  to  be  correlated  (Field,  2009).  In  neither  case,  did  the 
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principal  component  analysis  extraction  find  the  same  groupings  as  the  Oehmen  study. 
Therefore,  additional  review  was  required. 

The  questions  were  reexamined  for  both  face  and  content  validity.  The  Oehmen 
study  had  a  large  number  of  questions  that  were  deemed  to  be  exploratory  in  nature  and 
were  removed  from  further  statistical  analysis.  Given  the  small  sample  size,  the 
questions  were  reviewed  to  further  limit  the  dependent  variables  and  questions  to  fewer 
than  what  was  expected  at  the  beginning  of  this  study.  The  review  for  face  validity 
sought  to  ensure  the  questions  were  sound  and  met  a  basic  test  of  reason.  The  following 
content  validity  sought  to  ensure  that  the  questions  met  with  the  expectation  provided  by 
the  leading  risk  management  framework  reviewed  in  Chapter  2.  This  review  reduced  the 
survey  question  variables  from  173  questions  to  just  twenty. 

This  additional  review  produced  six  dependent  variables  each  with  a  number  of 
subset  questions.  Table  16,  below,  provides  the  six  variables  and  the  questions  associated 
with  each. 


Table  16.  Independent  Factors  and  Contents 


Independent  Variable  Factor 

Content  of  Variable 

RiskPlanning 

Questions  31,  35,  and  45 

RiskEvaluation 

Questions  85  -  87,  89  -  92 

RiskMonitor 

Questions  132,  133,  and  146 

Riskinfluence 

Questions  148,150,  and  151 

RiskPerception 

Questions  156,  158,  159,  and  160 

Since  the  same  factors  used  by  Oehmen  could  not  be  constructed,  the  variables 
were  changed.  While  some  of  the  same  questions  are  used  in  both  this  study  and 
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Oehmen’s,  there  isn’t  enough  overlap  to  compare  the  factors  directly.  Additional  detail 
on  the  comparison  can  be  found  in  the  results  for  the  third  research  question  found 
toward  the  end  of  this  chapter  during  the  discussion  for  the  third  research  question 
comparing  industry  and  Air  Force  practices.  From  here  on,  only  the  new  independent 
variable  names  will  be  used  for  clarity. 

The  first  variable  is  RiskPlanning.  The  three  questions  that  comprise  this  factor 
ask  about  employee  motivation  to  perform  risk  management  (Q3 1),  if  the  risk 
management  is  systematic  and  structured  (Q35),  and  if  risk  management  is  effectively 
integrated  with  other  PM  processes  (Q45).  The  questions  that  form  this  factor  are 
presented  in  the  section  of  the  survey  on  risk  management  planning.  If  project  team 
members  are  more  motivated  to  carry  out  risk  management  and  the  RM  processes  are 
better  integrated  with  the  rest  of  the  process,  one  might  expect  better  formulated  and 
implemented  risk  management  plans. 

The  second  variable  is  RiskEvaluation.  This  factor  includes  seven  questions  that 
all  appear  in  the  survey  section  on  evaluating  and  making  use  of  the  risk  analysis.  The 
questions  ask  respondents  about  how  well  risk  is  included  in  making  decisions.  For 
example,  students  are  asked  in  this  section  if  go/no-go  decisions  based  on  risk 
assessments  and  are  the  results  of  the  risk  analysis  used  in  making  cost/schedule/technical 
trade-offs.  One  might  expect  that  decisions  made  with  the  inclusion  of  risk  analysis  and 
assessments  should  lead  to  project  performance  that  more  closely  matches  the 
expectations. 

The  third  independent  variable  is  RiskMonitor.  The  three  questions  that  comprise 
this  factor  ask  respondents  about  their  project’s  risk  monitoring  and  review  process.  For 
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example,  respondents  were  asked  if  risk  were  escalated  to  senior  management  and  if  the 
risks  were  regularly  reviewed.  One  might  expect  that  risks  reviewed  and  discussed  with 
senior  managers  more  often  would  lead  to  increased  project  success. 

The  fourth  factor  is  Riskinfluence.  These  three  questions  ask  respondents  how 
much  influence  risk  management  has  in  their  organizations.  For  example,  these  questions 
ask  respondents  if  risk  management  processes  are  the  primary  mechanism  to  determine 
project  reserves  and  whether  the  findings  from  the  risk  management  processes  translate 
into  action.  One  might  reason  that  the  more  favorably  that  risk  management  is  perceived, 
the  more  risk  management  principles  might  be  considered  in  making  decisions  and 
allocating  resources. 

The  fifth  and  final  independent  factor  is  RiskPerception.  These  four  questions  ask 
respondents  about  how  risk  management  is  perceived  in  their  organization.  For  example, 
respondents  are  asked  if  they  believe  risk  management  creates/protects  value  and  if  risk 
management  has  a  positive  influence  on  program  success.  One  might  expect  that 
organizations  that  have  a  more  positive  perception  of  risk  management  might  be  expected 
to  internalize  the  recommendations  of  the  frameworks  reviewed. 

The  resulting  factor  analysis  was  performed  using  principal  component  analysis 
extraction  with  varimax  orthogonal  rotation.  Table  17  below  shows  the  variance 
explained  by  the  factors  created.  The  five  aggregated  factors  that  resulted  from  this 
analysis  are  presented  in  the  five  component  columns  (one  through  five)  with  the 
individual  questions  in  the  column  on  the  left.  The  individual  question  numbers  appear 
on  the  left  side  of  the  variable  name  (e.g.,  31)  and  the  mapping  to  the  original  Oehmen 
study  is  provided  on  the  right  side  (e.g.,  1.1).  The  communality  of  these  individual 
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questions  will  be  shown  to  be  similar  by  measuring  and  looking  for  components  with 
eigenvalues  greater  than  1.000. 

Table  17,  below,  displays  five  components  with  eigenvalues  greater  than  1.000. 


Table  17.  Factor  Analysis  Explained  Variance 


Component 

Initial  Eigenvalues 

Extraction  Sums  of  Squared 

Loadings 

Rotation  Sums  of  Squared 

Loadings 

%of 

Cumulative 

%of 

Cumulative 

%of 

Cumulative 

Total 

Variance 

% 

Total 

Variance 

% 

Total 

Variance 

% 

1 

8.894 

44.468 

44.468 

8.894 

44.468 

44.468 

4.799 

23.993 

23.993 

2 

2.491 

12.454 

56.922 

2.491 

12.454 

56.922 

3.430 

17.148 

41.141 

3 

1.542 

7.712 

64.634 

1.542 

7.712 

64.634 

2.488 

12.439 

53.580 

4 

1.163 

5.813 

70.447 

1.163 

5.813 

70.447 

2.379 

1 1 .894 

65.474 

5 

1.127 

5.633 

76.081 

1.127 

5.633 

76.081 

2.121 

10.607 

76.081 

6 

.614 

3.069 

79.149 

7 

.570 

2.850 

81.999 

8 

.527 

2.633 

84.633 

9 

.468 

2.338 

86.970 

10 

.418 

2.090 

89.060 

11 

.389 

1.943 

91.003 

12 

.318 

1.588 

92.591 

13 

.284 

1.421 

94.012 

14 

.265 

1.324 

95.336 

15 

.218 

1.090 

96.426 

16 

.193 

.965 

97.391 

17 

.178 

.888 

98.278 

18 

.127 

.633 

98.912 

19 

.120 

.602 

99.514 

20 

.097 

.486 

100.000 

Extraction  Method:  Principal  Component  Analysis. 


The  communality  between  the  questions  that  comprise  each  factor  is  summarized 
in  Table  18  below. 
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Table  18.  Rotated  Component  Matrix 


Question/Variable  Name 

Component  | 

1 

2 

3 

4 

5 

31  Risk  execution  1.1 

.776 

35  Risk  execution  5.9 

.763 

45  Risk  Process  Application  2.5 

.812 

85  Decision  Making 

.697 

86  Decision  Making  4.1 

.689 

87  Decision  Making 

.758 

89  Decision  Making  4.3 

.677 

90  Decision  Making 

.840 

91  Decision  Making 

.828 

92  Decision  Making 

.861 

132  RM  monitor  and  review  5.1 

.415 

.795 

133  RM  monitor  and  review  5.2 

.782 

146  Risk  management  performance 

.662 

148  Risk  management  performance 

.702 

150  Risk  management  performance 

.856 

151  Risk  management  performance 

.620 

156  Influence  of  risk  management 

.858 

158  Influence  of  risk  management 

.815 

159  Influence  of  risk  management 

.797 

160  Influence  of  risk  management 

.882 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Varimax  with  Kaiser  Normalization. 

Note:  Rotation  converged  in  7  iterations. 


This  analysis  produced  neatly  grouped  elements  into  five  components.  These 
groupings  confirmed  the  communality  of  the  content  questions  used  in  the  factors 
described  above.  There  were  a  few  that  fit  into  more  than  one  factor,  however.  Question 
132  shows  communality  with  both  the  first  and  the  third  factor  but  will  be  placed  on  the 
third  factor  for  further  analysis  because  it  shows  greater  communality  with  that  factor. 
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Next,  a  reliability  analysis  was  performed  to  confirm  that  the  questions/variables 
extracted  above  meet  a  standard  test  for  reliability.  In  this  sense,  reliability  refers  to  the 
repeatability  of  the  survey.  In  other  words,  if  one  were  to  take  and  retake  the  same 
survey,  a  reliable  survey  will  return  consistent  answers.  Measured  as  Cronbach’s  Alpha, 
a  few  authors  suggest  that  an  Alpha  greater  than  0.7  indicates  an  acceptable  result 
(George  and  Mallery,  2003;  Kline,  1999).  In  this  case,  SPSS  returns  a  Cronbach’s  Alpha 
of  0.936;  well  above  the  0.7  standard  recommended  as  “acceptable.”  Based  on  this,  one 
could  accept  that  the  variables  extracted  meet  the  test  of  reliability. 


Table  19.  Reliability  Analysis  (Cronbach's  Alpha) 


Cronbach's 

Alpha 

N  of  Items 

.936 

21 

With  a  favorable  reliability  analysis  result,  the  factors  were  aggregated  using  an 
average  of  their  component  question  scores.  Since  all  the  questions  within  each  factor 
were  answered  on  the  same  scale,  no  additional  adjustment  was  required.  Next,  the 
descriptive  statistics  of  each  aggregated  factor  were  reviewed  to  ensure  normality.  This 
was  performed  by  examining  the  skewness  and  kurtosis  of  each  factor.  The  descriptive 
statistics  are  presented  in  Table  20  below.  One  heuristic  suggests  that  one  test  for 
normality  would  be  a  skewness  and  kurtosis  statistic  between  -1  and  1.  In  this  case,  all 
factors  appear  to  meet  this  standard  of  normality,  although  Technical  Performance 
Success,  Riskinfluence,  RiskMonitor,  and  RiskPerception  do  have  slightly  elevated 
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statistics  for  skewness  and/or  kurtosis,  which  may  indicate  non-normality  of  the  data  in 


use. 


Table  20.  Aggregated  Factors  and  Dependent  Variables  Descriptive  Statistics* 


N 

Minimum 

Maximum 

Mean 

Std. 

Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Std. 

Error 

Statistic 

Std. 

Error 

Cost  Success 

82 

1 

5 

2.98 

.875 

-.066 

.266 

.737 

.526 

Schedule  Success 

82 

1 

5 

2.93 

1.016 

.149 

.266 

-.254 

.526 

Technical  Perf. 

82 

1 

5 

3.06 

.654 

-.062 

.266 

2.868 

.526 

Success 

Customer  Sat. 

81 

1 

5 

3.04 

.782 

-.065 

.267 

1.248 

.529 

Success 

RiskPlanning 

118 

1.00 

5.00 

3.3305 

.90319 

-.437 

.223 

.196 

.442 

RiskEvaluation 

92 

1.00 

5.00 

3.0140 

.89529 

-.374 

.251 

.016 

.498 

RiskMonitor 

88 

1.00 

5.00 

3.7765 

.83003 

-1.113 

.257 

1.345 

.508 

Riskinfluence 

90 

1.00 

5.00 

3.2519 

.87881 

-.590 

.254 

.472 

.503 

RiskPerception 

90 

1.00 

5.00 

3.6722 

.85101 

-.844 

.254 

1.061 

.503 

Valid  N  (listwise) 

76 

Next,  a  correlation  table  is  presented  displaying  the  covariance  between  the  five 
independent,  aggregated  factor  variables  and  the  four  dependent,  outcome  variables. 
Table  21  summarizes  these  correlations  below. 

These  correlations  provide  some  insight  to  answering  the  second  and  third  research 
questions.  All  five  of  the  factors  are  positively  correlated  with  the  outcome  variables  and 
all  are  significantly  correlated  with  at  least  one  outcome  variable.  All  four  dependent 


'  The  statistics  for  skewness  and  kurtosis  indicate  the  data  in  use  may  violate  the  assumption  of  normality 
required  for  the  use  of  ordinary  least  squares  (OLS)  regression.  The  same  regressions  presented  later  were 
also  performed  using  ordinal  regression  which  doesn’t  require  the  assumption  of  normality.  No  material 
change  was  found  between  the  two  different  regression  types  so  the  OLS  regressions  are  presented  alone 
for  clarity. 
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variables  exhibited  statistically  significant  correlations.  From  a  practical  standpoint,  all 
four  of  these  variables  are  also  highly  correlated.  These  relationships  suggest  that  there 
really  is  a  strong  interdependence  between  cost,  schedule,  performance,  and  in  this  case, 
customer  satisfaction. 


Table  21.  Independent  and  Dependent  Variable  Correlations 


Tech. 

Cust. 

Risk 

Risk 

Risk 

Risk 

Cost 

Sched 

Perf. 

Sat. 

Eval. 

Monitor 

Influence 

Percept. 

Succ. 

Succ. 

Succ 

Succ 

Risk 

Pearson 

1 

.511 

.517 

.450 

.397 

.044 

.149 

.153 

.284 

Planning 

Corr. 

Sig.  (2- 

.000 

.000 

.000 

.000 

.694 

.183 

.170 

.010 

tailed) 

N 

118 

92 

88 

90 

90 

82 

82 

82 

81 

Risk 

Pearson 

.511 

1 

.616 

.485 

.424 

.117 

.263 

.309 

.294 

Eval. 

Corr. 

Sig.  (2- 

.000 

.000 

.000 

.000 

.306 

.019 

.006 

.009 

tailed) 

N 

92 

92 

85 

85 

86 

79 

79 

79 

78 

Risk 

Pearson 

.517 

.616 

1 

.463 

.398 

.137 

.223 

.323 

.271 

Monitor 

Corr. 

Sig.  (2- 

.000 

.000 

.000 

.000 

.227 

.048 

.004 

.016 

tailed) 

N 

88 

85 

88 

86 

87 

79 

79 

79 

79 

Risk 

Pearson 

.450 

.485 

.463 

1 

.549 

.143 

.165 

.212 

.244 

Influence 

Corr. 

Sig.  (2- 

.000 

.000 

.000 

.000 

.199 

.139 

.056 

.028 

tailed) 

N 

90 

85 

86 

90 

89 

82 

82 

82 

81 

Risk 

Pearson 

.397 

.424 

.398 

.549 

1 

.038 

.190 

.404 

.471 

Percept. 

Corr. 

Sig.  (2- 

.000 

.000 

.000 

.000 

.734 

.087 

.000 

.000 

tailed) 

N 

90 

86 

87 

89 

90 

82 

82 

82 

81 

Note:  Significant  correlation  at  the  0.05  ievel  (2-taiied)  is  noted  in  boid  face. 


None  of  the  aggregated  factors  correlate  with  the  cost  success  variable.  This 
finding  matches  the  Oehmen  study  which  also  found  no  questions  that  significantly 
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correlated  with  cost  success.  Since  these  factors  are  compromised  of  the  same  individual 
questions,  this  is  of  little  surprise. 

Two  of  the  factors  correlate  with  schedule  success.  They  are  RiskE valuation  and 
RiskMonitor.  Recall  that  these  two  variables  capture  how  well  risk  analysis  is 
incorporated  into  decision  making  and  how  the  risks  are  monitored  and  reviewed.  This 
finding  provides  support  for  the  conclusion  that  a  relationship  does  exist  between  risk 
management  and  project  success  outcomes. 

Three  factors  correlate  positively  with  project  performance  success. 
RiskEvaluation,  RiskMonitor,  and  RiskPerception  are  all  shown  to  have  a  relationship 
with  this  outcome  variable.  The  first  two  factors  were  discussed  previously  with  their 
relationship  to  schedule  success.  RiskPerception  attempts  to  capture  how  the 
organization  perceives  the  value  of  risk  management.  These  findings  also  suggest  that  a 
relationship  exists  between  these  risk  management  practices  and  project  success. 

Finally,  all  five  independent  factors  are  shown  to  have  a  significant  relationship 
with  customer  satisfaction.  It’s  interesting  that  all  five  variables  are  shown  to  relate  to 
this  outcome  variable  as  compared  to  the  other  three  outcome  variables.  This  finding 
lends  support  that  a  relationship  exists  between  risk  management  and  project  success. 
Regression  Models 

Correlation  is  interesting  is  not  as  powerful  as  more  rigorous  methods  such  as 
regression.  Four  multiple  linear  regression  models  examine  more  deeply  the  relationship 
between  these  five  independent  factors  and  the  four  dependent  variables.  A  multiple 
linear  regression  was  performed  for  each  outcome  with  all  five  independent  factors 
included.  The  results  of  each  multiple  linear  regression  are  provided  below. 
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Cost  Success  Regression 

The  cost  outcome  was  analyzed  first.  The  model  summary  is  provided  in  Table 
22  below. 


Table  22.  Cost  Regression  R-Square  Values 


Model 

R 

R  Square 

Adjusted  R  Square 

Std.  Error  of  the  Estimate 

1 

.232 

.054 

-.013 

.878 

With  an  R-square  of  0.054,  this  model  does  not  appear  to  offer  a  great  deal  of 
insight  into  what  controls  cost.  Based  on  the  correlation  results  presented  in  Table  21 
above,  this  result  is  unsurprising.  That  table  revealed  no  independent  factors  to  correlate 
with  cost  success. 

In  addition  to  not  providing  much  insight  into  the  variance  of  cost  success,  it  also 
fails  to  provide  a  statistically  significant  model  as  shown  by  the  0.548  significance  in 
Table  23  below. 


Table  23.  Cost  Regression  ANOVA  Results 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

3.112 

5 

.622 

.808 

.548 

Residual 

54.680 

71 

.770 

Total 

57.792 

76 

Finally,  the  multiple  linear  regression  model  for  cost  success  is  shown  below  in 
Table  24. 
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Table  24.  Linear  Regression  Cost  Success  Model 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

2.479 

.544 

4.558 

.000 

RiskPlanning 

-.062 

.140 

-.063 

-.446 

.657 

RiskEvaluation 

.059 

.146 

.062 

.406 

.686 

RiskMonitor 

.076 

.160 

.073 

.473 

.637 

Risklnfluence 

.231 

.150 

.233 

1.541 

.128 

RiskPerception 

-.148 

.148 

-.150 

-1.000 

.321 

As  expected,  no  variables  appear  as  significant  influences  to  cost  success.  The 
variable  closest  to  significance,  Risklnfluence,  has  a  moderate  parameter  estimate  but 
wouldn’t  even  be  significant  at  the  0.10  level. 

Schedule  Success  Regression 

Next,  the  impact  of  risk  management  on  schedule  success  was  analyzed.  This 
model  seeks  to  explain  the  influence  that  risk  management  has  on  a  given  project  meeting 
its  schedule  success  criteria. 

Table  25,  below,  shows  the  schedule  model’s  summary. 


Table  25.  Schedule  Regression  R-Square  Values 


Model 

R 

R  Square 

Adjusted  R  Square 

Std.  Error  of  the  Estimate 

1 

.280 

.078 

.013 

1.009 

The  R-Square  value  of  0.078  for  the  schedule  model  indicates  the  model  explains 
only  slightly  more  of  the  variance  of  schedule  success  than  it  does  for  cost  success.  This 
also  indicates  that  the  additional  influences  for  schedule  success  remain  unexplained. 
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Table  26,  below,  shows  the  ANOVA  table  for  the  schedule  success  model. 


Table  26.  Schedule  Regression  ANOVA  Results 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

mm 

1  Regression 

6.126 

5 

1.225 

1.204 

.316 

Residual 

72.238 

71 

1.017 

Total 

78.364 

76 

As  indicated,  the  model  shows  no  significance.  In  combination  with  the  R-Square 
above,  this  model  may  not  have  much  explanatory  insight  for  describing  any  relationship 
between  risk  management  and  schedule  success. 

Below  in  Table  27  is  the  linear  regression  model  for  schedule  success. 


Table  27.  Linear  Regression  Schedule  Success  Model 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.675 

.624 

2.684 

.009 

RiskPlanning 

-.028 

.162 

-.024 

-.175 

.862 

RiskEvaluation 

.201 

.167 

.183 

1.202 

.233 

RiskMonitor 

.082 

.183 

.069 

.451 

.654 

Riskinfluence 

.066 

.173 

.057 

.383 

.702 

RiskPerception 

.053 

.170 

.046 

.314 

.755 

In  this  case,  none  of  the  independent  variables  are  statistically  significant. 
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Technical  Performance  Success  Regression 

The  third  linear  regression  model  seeks  to  explain  how  the  risk  management 
model  influences  the  project  meeting  its  technical  performance  success  measures.  First, 
the  model  summary  for  the  linear  regression  model  predicting  technical  performance  is 
detailed  in  Table  28  below. 


Table  28.  Technical  Performance  R-Square  Value 


Model 

R 

R  Square 

Adjusted  R  Square 

Std.  Error  of  the  Estimate 

1 

.453 

.205 

.149 

.615 

With  an  R-square  of  0.205,  this  model  offers  a  bit  more  insight  into  the  technical 
performance  success  than  it  did  for  the  cost  and  schedule  success  models.  Again,  though, 
the  majority  of  the  variance  here  still  remains  unexplained. 

Next,  the  ANOVA  results  are  provided  in  Table  29  below. 


Table  29.  Technical  Performance  Regression  ANOVA  Results 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

6.929 

5 

1.386 

3.663 

.005 

Residual 

26.863 

71 

.378 

Total 

33.792 

76 

Unlike  the  previous  models,  this  model  is  significant  with  a  p-value  of  0.005. 
Finally,  the  multiple  linear  regression  model  for  technical  performance  success  is 
provided  in  Table  30  below. 
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Table  30.  Linear  Regression  Technical  Performance  Success  Model 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.715 

.381 

4.499 

.000 

RiskPlanning 

-.095 

.098 

-.125 

-.969 

.336 

RiskEvaluation 

.094 

.102 

.129 

.916 

.363 

RiskMonitor 

.143 

.112 

.180 

1.273 

.207 

Riskinfluence 

-.044 

.105 

-.058 

-.415 

.679 

RiskPerception 

.266 

.103 

.352 

2.568 

.012 

This  model  does  feature  a  significant  variable.  RiskPerception  is  found  to  be 
significant  with  a  p-value  of  0.012.  This  suggests  that  the  organization’s  perception  of 
the  value  of  risk  management  has  a  significant  and  moderately  positive  influence  on 
project  technical  performance  success.  This  provides  support  for  the  existence  of  a 
relationship  between  an  element  of  risk  management  and  an  element  of  project  success. 

Customer  Satisfaction  Success  Regression 

The  final  model  seeks  to  explain  the  influence  the  risk  management  model  has  on 
the  final  dependent  variable,  customer  satisfaction.  The  model  summary  for  customer 
satisfaction  success  is  presented  in  Table  31  below. 


Table  3 1 .  Customer  Satisfaction  Regression  R-Square  Value 


Model 

R 

R  Square 

Adjusted  R  Square 

Std.  Error  of  the  Estimate 

1 

.491 

.241 

.187 

.685 

77 


This  model  seems  to  provide  more  insight  into  explaining  the  variance  of 
customer  satisfaction  than  it  did  for  the  other  outcome  variables.  In  this  case,  the  model 
explains  almost  25%  of  the  variance  of  customer  satisfaction  success. 

Next,  the  ANOVA  results  are  presented  in  Table  32  below. 


Table  32.  Customer  Satisfaction  Regression  ANOVA  Results 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

mm 

1  Regression 

10.586 

5 

2.117 

4.506 

.001 

Residual 

33.362 

71 

.470 

Total 

43.948 

76 

The  ANOVA  results  provided  indicate  that  the  model  is  highly  significant  with  a 
p- value  of  0.001. 

Table  33,  below,  provides  the  table  for  the  customer  satisfaction  success  linear 
regression. 


Table  33.  Linear  Regression  Customer  Satisfaction  Model 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.326 

.425 

3.121 

.003 

RiskPlanning 

.062 

.109 

.072 

.570 

.570 

RiskEvaluation 

.054 

.114 

.066 

.477 

.635 

RiskMonitor 

.022 

.125 

.024 

.173 

.863 

Riskinfluence 

-.032 

.117 

-.036 

-.269 

.788 

RiskPerception 

.369 

.115 

.430 

3.204 

.002 
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Once  again,  RiskPerception  is  found  to  be  a  significant  variable  in  explaining  a 
project  success  outcome.  In  this  model,  RiskPerception  is  highly  significant  and 
moderately  impactful.  This  provides  additional  information  to  support  the  notion  that 
risk  management  does  have  a  relationship  with  customer  satisfaction  and  is  effective  at 
aiding  the  project  in  meeting  its  outcome  goals. 

In  summary,  the  models  provide  some  insight  into  explaining  risk  management 
effectiveness  and  its  impact  on  project  performance.  However,  based  on  the  R-Square 
values  and  few  significant  variables,  much  remains  unexplained.  It  does  appear  that 
RiskPerception  has  a  positive  and  significant  influence  on  two  of  the  four  outcome 
variables.  However,  the  model  tested  provides  little  insight  into  the  relationship  or 
effectiveness  of  risk  management  with  respect  to  cost  and  schedule  success. 

Research  Question  3:  Correlation  Analysis  Comparison 

The  third  research  question  asks  if  a  difference  exists  between  the  findings  for  this 
study  and  the  findings  for  Oehmen  study.  Recall  that  there  are  some  challenges  in 
making  a  direct  comparison  here.  Recall,  also,  that  the  Oehmen  study  selected  an 
additional  nine  outcome  variables  that  were  considered  as  intermediate  steps,  not 
outcome  variables,  for  this  study.  These  two  studies  use  different  measures  of 
correlations  and  different  levels  of  significance.  Still,  this  comparison  could  be 
insightful. 

Table  34,  below,  duplicates  a  previous  table  and  provides  the  questions  that  the 
Oehmen  study  found  to  correlate  significantly  with  schedule  or  customer  satisfaction 
success.  Recall  that  the  Oehmen  study  found  no  questions  that  correlated  significantly 
with  either  cost  or  technical  performance  success. 
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Table  34.  Oehmen  Significant  Variables 


Variable 

Schedule  Success 
(Kruskal-Goodman 
Gamma) 

Customer 

Satisfaction  Success 
(Kruskal-Goodman 
Gamma) 

Q3 1  Our  employees  are  motivated  to 
perform/implement  risk  management(RM) 

- 

0.43 

Q34  Our  RM  explicitly  addresses  uncertainty 

- 

0.42 

Q35  Our  RM  is  systematic,  structured  and  timely 

- 

0.42 

Q38  Our  RM  takes  human  and  cultural  factors  into 
account 

- 

0.41 

Q39  Our  RM  is  transparent  and  inclusive  towards 
all  stakeholders 

- 

0.51 

Q40  Our  RM  is  dynamic,  iterative  and  responsive 
to  change 

- 

0.40 

Q45  The  RM  process  is  effectively  integrated. . . 

- 

0.46 

Q78a  The  impacts  of  risk  are  quantified  using  cost 
as  a  dimension^ 

0.41 

- 

Q78b  The  impacts  of  risk  are  quantified  using 
technical  performance. .  .as  a  dimension^ 

0.49 

- 

Q86  Resources  are  allocated  to  reduce  largest  risks 
as  early  as  possible 

- 

0.42 

(Oehmen,  et.al,  2014) 


The  previous  correlation  analysis  for  Research  Question  1  finds  52  questions  that 
correlate  with  the  four  dependent  variables.  This  contrasts  with  a  list  of  only  eight  from 
the  Oehmen  study  that  correlate  with  the  four  dependent  variables  from  this  study.  Of 
those  questions,  the  two  studies  find  four  questions  in  common.  Table  35,  below, 
summarizes  these  similar  findings. 


^  This  question  is  one  of  the  qualitative  questions  not  used  for  this  study. 
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Table  35.  Shared  Significant  Questions 


Common  Variable 

Air  Force  Sample 
(Pearson) 

Cost  Schedule  Customer 

Success  Success  Satisfaction 

Oehmen  Study 
(K-G  Gamma) 

Technical  Customer 

Performance  Satisfaction 

Q3 1  Our  employees  are 
motivated  to 
perform/implement  risk 
management(RJV[) 

- 

- 

0.295 

(0.008) 

- 

0.43 

Q35.  RM  is  systematic, 
structured  and  timely 

- 

- 

0.238 

(0.032) 

- 

0.42 

Q45.  RM  teams  are 
cross-functional  and 
cross-organizational 

- 

0.236 

(0.033) 

0.268 

(0.015) 

- 

0.46 

There  are  a  number  of  interesting  findings  from  these  results.  The  RM  activities 
associated  with  questions  31,  35,  and  45  are  found  to  correlate  significantly  with 
customer  satisfaction  in  both  studies.  Question  45  is  also  found  to  correlate  with 
schedule  success  by  this  study. 

While  this  study  and  the  Oehmen  study  found  different  factors,  there  were  still 
some  similarities.  A  comparison  is  provided  in  Table  36  below.  As  the  table  notes, 
there  is  not  a  great  deal  of  overlap  between  the  two  studies  with  respect  to  factors.  Two 
of  the  factors  (RiskEvaluation  and  RiskJVtonitor)  are  similar  enough  to  factors  used  by  the 
Oehmen  study  that  they  might  be  considered  reasonably  similar.  The  remaining  three 
constructed  factors  for  this  study  share  little  with  the  other  Oehmen  factors. 

Riskinfluence  and  RiskPerception  share  one  question  each  from  the  Oehmen  factors 
while  RiskPlanning  share  three  questions  but  from  three  different  Oehmen  factors.  In 
short,  no  conclusion  can  be  drawn  from  this  factor  analysis. 
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Table  36.  Independent  Variable  Comparison 


New  Independent  Variable 
Factor 

Similar  Oehmen 
Independent  Variable 

Shared  Questions 

IV I .  RiskPlanning 

Shares  Questions  31,  35,  and 
45  (Oehmen’s  1.1,  5.9,  and 
2.5) 

IV2.  RiskEvaluation 

4.  Support  all  Critical 
Decisions  with  Risk 
Management  Results 

Shares  Questions  86  and  89 
(Oehmen’s  4.1  and  4.3) 

5.  Monitor  and  Review  Your 

Shares  Questions  132,  133, 

IV3.  RiskMonitor 

Risks,  Risk  Mitigation 
Actions,  and  RM  Process 

and  146  (Oehmen’s  5.1,  5.2, 
and  C.4.) 

IV4.  Riskinfluence 

- 

Shares  Question  148 
(Oehmen’s  A.2.) 

IV5.  RiskPerception 

- 

Shares  Questions  158 
(Oehmen’s  A.l.) 

Investigative  Questions  Answered 

This  study  looks  at  the  relationship  between  risk  management  and  project  success. 
It  additionally  seeks  to  determine  if  the  risk  management  efforts  are  effective  and  if 
specific  practices  that  relate  to  project  success  can  be  isolated.  The  research  questions 
and  their  associated  hypotheses  are  presented  in  Table  37  below. 

Research  Question  1 

The  first  research  question  asks  if  a  relationship  exists  between  risk  management 
and  project  success  outcomes.  A  number  of  correlation  tables  were  reviewed  containing 
results  from  individual  questions  and  their  relationship  with  the  four  dependent  outcome 
variables.  Those  questions  were  mapped  to  the  risk  management  frameworks  reviewed. 
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Table  37.  Research  Questions  and  Hypotheses 


Question 

Hypothesis 

Q 1 .  Does  a  relationship  exist  between  risk 
management  and  an  Air  Force  project 
meeting  its  cost,  schedule,  and  performance 
goals? 

HI.  There  exists  a  statistically  significant 
correlation  between  risk  management 
practices  and  Air  Force  project  success 
relative  to  cost,  schedule  or  performance 
goals. 

Q2.  How  effective  is  the  Air  Force  risk 
management  effort  in  aiding  its  projects  to 
meet  their  given  cost,  schedule  and 
performance  targets? 

H2.  At  least  one  risk  management  practice 
will  be  shown  to  significantly  affect  Air 
Force  project  success  relative  to  cost, 
schedule  or  performance  goals. 

Q3.  Are  there  differences  between  the 
effective  risk  management  practices  used  in 
industry  and  those  used  by  the  Air  Force? 

H3.  A  difference  does  not  exist  between 
the  effective  risk  management  practices 
used  in  industry  and  those  used  by  the  Air 
Force. 

Questions  from  areas  pertaining  to  risk  planning,  risk  analysis,  risk  monitoring  and 
review,  risk  performance,  and  risk  mitigation  were  reviewed.  In  total,  46  questions  are 
found  to  correlate  significantly. 

Not  all  of  the  outcome  variables  showed  the  same  number  of  correlated  questions. 
The  cost  success  dependent  variable  has  no  correlated  questions.  The  schedule  success 
variables  had  a  few  more  significantly  correlated  variables  but  some  of  those,  too,  had  a 
negative  relationship.  The  other  two  dependent  variables,  technical  performance  success 
and  customer  satisfaction  success,  showed  the  greatest  number  of  questions  with  a 
significant  correlation. 

Additionally,  some  of  the  questions  asked  about  risk  reduction  achieved.  These 
questions  measure  impact  of  the  risk  mitigation(s)  employed  rather  than  the  planning  and 
analysis  done  to  prior  to  the  risk  occurring.  In  this  section,  respondents  indicated  there 
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were  eighteen  questions  that  positively  and  significantly  correlated  with  project  success 
outcomes.  While  none  correlated  with  cost  success,  these  questions  also  show  a 
relationship  between  risk  management  and  project  cost,  schedule,  and  technical 
performance  success. 

HI.  Answer:  Reject  the  null  hypothesis  and  conclude  that  a  statistically 
significant  correlation  does  exist  between  risk  management  practices  and  Air  Force 
project  success  relative  to  cost,  schedule  or  performance  goals. 

Research  Question  2 

Now  that  a  relationship  has  been  shown  to  exist  between  risk  management  and 
project  success  outcomes,  the  second  research  question  asks  how  effective  it  is.  Of  the 
46  questions  found  to  have  a  relationship  with  one  or  more  of  the  four  outcome  variables, 
all  but  eight  were  found  to  have  a  positive  relationship  with  those  outcome  variables. 
Additionally,  the  factor  analysis  that  was  performed  also  found  all  five  constructed 
factors  to  be  significantly  and  positively  correlated  with  at  least  one  dependent  variable. 
Finally,  the  regression  models  developed  and  tested  found  two  of  the  models  to  be 
significant  but  the  RiskPerception  factor  is  the  only  factor  found  to  have  a  significant 
relationship  with  any  of  the  dependent  variables.  RiskPerception  is  found  to  have  a 
positive  and  significant  relationship  with  technical  performance  and  customer  satisfaction 
success. 

H2.  Answer:  Reject  the  null  hypothesis  in  favor  of  the  alternate  hypothesis  and 
accept  that  one  or  more  risk  management  practices  are  found  to  relate  significantly  and 
positively  with  project  success. 


84 


Research  Question  3 

The  third  research  question  seeks  to  determine  if  the  effective  risk  practices  in  use 
by  industry  match  those  in  use  by  the  Air  Force.  A  comparison  between  the  industry 
study  performed  by  Oehmen,  et  al.  and  this  survey  study  found  a  number  of  common 
practices.  The  Oehmen  study  found  only  ten  specific  questions  to  correlate  significantly 
with  cost,  schedule,  technical  performance,  or  customer  satisfaction  success;  two  with 
schedule  success  and  eight  with  customer  satisfaction.  The  Air  Force  sample  for  this 
study  found  three  of  those  ten  to  also  correlate  significantly  and  in  the  same  direction 
(positive)  as  the  industry  sample. 

Answering  this  research  question  is  not  as  clear  as  answering  the  previous  two. 
There  is  some  overlap  but  not  a  significant  amount  of  overlap.  Increasing  the  confidence 
level  of  the  AF  study  would  reduce  the  number  of  significant  questions  from  46  to  26. 
However,  it  would  also  reduce  the  number  of  shared  questions  from  three  to  one.  The 
differences  in  results  and  methodologies  between  the  two  studies  prevent  a  conclusive 
finding  to  this  research  question.  This  study  fails  to  provide  conclusive  results  to  warrant 
rejection  of  the  null  hypothesis.  While  the  findings  here  are  less  conclusive,  the 
methodology  for  this  question  was  to  accept  the  null  if  it  could  not  be  rejected. 

H3.  Answer:  Accept  the  null  hypothesis  and  conclude  a  difference  exists  between 
the  industry  sample  and  the  Air  Force  sample. 

Summary 

This  chapter  summarizes  the  analysis  results  for  this  study.  A  correlation  analysis 
compared  the  individual  survey  questions  with  the  four  dependent  variables  and  found  a 
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relationship  does  exist  between  risk  management  practices  in  use  by  the  Air  Force 
acquisition  personnel  surveyed.  That  correlation  analysis  also  shows  a  great  deal  of 
overlap  between  the  practices  in  use  from  an  industry  survey  and  the  practices  in  use  by 
that  Air  Force  sample.  Additionally,  a  factor  analysis  is  used  to  assemble  four  multiple 
linear  regression  models  comparing  the  five  developed  factors  and  their  relationship  with 
cost,  schedule,  technical  performance,  and  customer  satisfaction  success.  Those  models 
also  confirmed  that  a  relationship  exists  between  the  risk  management  models  and  one  of 
the  constructed  factors  with  project  success. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  summarizes  the  findings  of  this  study  and  provides  additional  insight 
and  recommendations  for  future  efforts.  The  eonelusions  reached  by  the  study  are  that  a 
relationship  does  exist  between  risk  management  and  projeet  sueeess.  Furthermore,  that 
relationship  is  a  positive  one  where  risk  management  may  be  effective  at  aiding  the 
projeet  to  reaeh  its  cost,  schedule,  teehnieal  performance,  and  customer  satisfaetion 
targets.  These  findings  provide  support  to  a  variety  of  risk  management  frameworks  and 
their  prescriptions. 

While  the  findings  are  interesting,  there  are  still  some  limitations  to  this  study. 

The  small  sample  size  prevents  this  study  being  deelared  as  much  more  than  a  glimpse 
into  the  relationship  between  risk  management  and  project  success.  In  addition  to  the 
small  sample  size,  the  respondents  were  more  homogenous  than  what  would  be  desired  to 
allow  these  finding  to  be  generalized  more  broadly  than  the  small  population  the  surveys 
were  drawn  from.  However,  despite  the  small  sample  size,  the  findings  do  support 
reeommendations  for  aetion  and  future  research. 

However,  additional  steps  remain  to  fully  deseribe  the  relationship  between  risk 
management  and  project  success.  There  appears  to  be  a  relationship,  but  it  needs  to  be 
more  clearly  defined  beyond  mere  correlation.  In  order  to  better  define  this  relationship, 
risk  management  needs  to  be  more  elearly  separated  from  the  other  praetiees  within 
projeet  management  and  systems  engineering. 
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In  the  meantime,  additional  recommendations  can  be  made.  The  risk 
management  frameworks  in  use  and  their  prescriptions  for  a  number  of  specific  practices 
do  appear  to  have  a  positive  relationship.  What  is  also  interesting  is  the  organizational 
behavior  aspect  of  risk  management  that  appears  to  have  a  significant  and  positive 
relationship  with  project  success;  specifically,  customer  satisfaction.  Policy  makers  and 
educators  of  risk  management  should  consider  this  effect. 

Conclusions  of  Research 

Before  reviewing  the  conclusions,  a  number  of  assumptions  and  limitations 
should  be  reviewed  to  temper  any  conclusions  from  this  study.  For  one,  all  of  the  survey 
responses  are  self-reported  including  the  outcome  or  dependent  variables.  Additionally, 
all  of  this  data  is  provided  by  those  serving  in  acquisition  assignments  and  not  in  the  roles 
of  the  customer  themselves.  Because  those  in  the  acquisition  position  may  have  better 
insight  into  project  performance  for  a  window  in  time,  this  self-reported  information  is 
still  useful.  Additionally,  since  the  sample  is  likely  comprised  entirely  of  acquisition 
program  managers  who  are,  by  DoD  policy,  most  responsible  for  risk  management,  the 
results  here  may  provide  an  insight  into  risk  management  across  the  Air  Force  acquisition 
system. 

As  described  in  the  conclusions  from  Chapter  4,  risk  management  appears  to  have 
a  positive  relationship  with  project  performance  at  least  within  the  minds  of  the 
respondents.  46  specific  practices  from  the  four  tested  steps  in  a  notional  risk 
management  framework  are  found  to  be  significantly  correlated  with  project  success. 

This  may  confirm  that  a  relationship  exists  for  this  sample  but  any  generalization  beyond 
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this  group  requires  the  assumption  that  this  sample  is  representative  of  all  Air  Force 
acquisition  personnel  —  an  assumption  that  cannot  be  tested  and  proven. 

In  addition  to  confirming  the  presence  of  a  relationship,  this  study  showed  that 
risk  management  may  be  effective  at  aiding  projects  in  meeting  their  cost,  schedule, 
technical  performance,  and  customer  satisfaction  goals.  While  correlation  does  not  prove 
causation,  it’s  still  worth  noting  that  all  46  of  the  correlated  questions  have  a  positive 
relationship  with  risk  management. 

There  are  a  number  of  potential  explanations  for  this  lack  of  questions  that 
correlated  with  cost  success.  The  contract  type  in  use  could  be  one  example.  Because 
the  DoD  currently  has  a  preference  for  using  fixed-price  contracts  in  many  cases,  this 
sample’s  respondents  might  all  be  working  with  their  contractor  via  a  fixed  price 
contract.  In  this  case,  any  cost  savings  or  reductions  would  accrue  to  the  developer  and 
not  to  the  government.  One  other  possible  explanation  is  the  priorities  in  place  in  DoD 
acquisition  and,  likely,  the  AF  as  well.  In  a  number  of  cases,  the  DoD  seems  more 
willing  to  tolerate  cost  overruns  while  less  willing  to  tolerate  schedule  and  technical 
performance  problems.  This  finding  could  be  the  result  of  the  intentional  trade  of  cost  for 
performance  or  time. 

The  correlation  analysis  also  provides  some  support  to  the  risk  management 
frameworks  in  use  in  a  variety  of  organizations.  Specific  risk  management  practices  were 
found  to  positively  and  significantly  influence  all  four  of  the  tested  risk  management 
framework  steps  (no  attempt  was  made  to  analyze  risk  identification).  This  provides 
some  validation  to  the  basic  risk  management  model. 
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There  are  plenty  of  examples  here  that  offer  some  support  for  the  inelusion  of  risk 
analysis  in  the  project’s  decision  making  process.  The  trade-offs  and  interdependence 
between  cost,  schedule,  and  performance  has  been  noted.  This  list  provides  numerous 
examples  of  significantly  correlated  questions  that  indicate  that  considering  risk  is 
positively  covariant  with  a  successful  project.  Since  risk  analysis  comprises  a  major 
portion  of  the  risk  frameworks  reviewed,  this  also  offers  some  support  for  the  continued 
inclusion  of  risk  management  analysis  in  project  decision  making. 

What’s  also  interesting  is  the  covariance  revealed  between  organizational 
behavior  issues  and  project  success.  None  of  the  frameworks  explicitly  call  for  a  free  and 
open  exchange  of  information  within  the  project  team;  though,  at  least  a  few  of  them 
imply  that  it  should  take  place  (DoD,  2013;  PMI,  2013).  Oehmen  also  notes  this 
important  part  of  risk  management  (Oehmen,  et  ah,  2014).  This  finding  suggests  that 
there  could  be  more  to  effective  risk  management  than  merely  completing  the  steps 
prescribed  within  a  given  framework. 

The  correlation  results  from  Chapter  4  show  that  a  relationship  does  appear  to 
exist  between  risk  management  and  the  four  project  success  variables.  Numerous 
questions  from  the  four  risk  management  steps  tested  (risk  planning,  risk  analysis,  risk 
mitigation  and  implementation,  and  risk  tracking)  proved  to  have  a  significant 
relationship.  This  provides  support  that,  not  only  does  the  relationship  exists,  but  that  it 
can  be  extended  to  at  least  the  individual  risk  management  step. 

The  factor  analysis  and  multiple  linear  regressions  also  provide  valuable  feedback 
on  risk  management.  The  factor  analysis  identified  five  broad  factors  that  are  tested  for 
their  relationship  with  the  four  dependent  outcome  variables.  While  these  factors  proved 
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to  be  different  than  the  Oehmen  study  factors,  they  still  provided  some  support  for  those 
factors  since  they  shared  a  number  of  individual  questions. 

Though  two  of  the  three  research  questions  could  be  conclusively  answered,  there 
are  still  some  limits  to  the  findings  of  this  study.  Since  this  study  relied  heavily  on 
correlation,  there  are  limits  to  the  conclusions  that  can  be  drawn  from  it.  The  factor 
analysis  and  multiple  linear  regressions  also  provide  some  insight  but  that  insight  is 
limited  by  the  survey  response  size  and  homogeneity  noted  above.  Most  seriously, 
perhaps,  all  of  the  results  are  the  opinion  of  the  respondent.  It’s  entirely  possible  that  the 
individuals  who  think  more  highly  about  risk  management  responded  more  positively 
about  its  impact.  Recall  that  the  only  factor  that  was  found  to  be  significant  in  any  of  the 
models  was  the  factor  that  represented  the  value  the  respondent  placed  on  risk 
management  (RiskPerception).  It’s  entirely  possible  that  those  with  a  high  opinion  of 
risk  management  simply  responded  that  it  was  effective  because  they  have  a  high  opinion 
of  it.  Note  that  the  two  regression  models  that  featured  RiskPerception  as  a  significant 
factor  were  the  two  most  that  are  most  subjectively  measured.  In  sum,  all  that  can  be 
conclusively  shown  is  that  certain  types  of  risk  management,  specifically  those  related  to 
Risk  Perception,  are  associated  in  the  minds  of  the  survey  respondents  with  more 
favorable  technical  performance  outcomes. 

Additionally,  despite  the  interesting  findings,  the  methodology  employed  still 
makes  it  difficult  to  fully  conclude  the  results  are  due  solely  to  risk  management.  Project 
management  and  systems  engineering  influences  may  also  play  a  role  in  project  success. 
Since  risk  management  is  included  in  both  of  those  sets,  the  results  here  may  be  conflated 
with  those  two  other  broader  sets  of  tools  and  practices.  Additionally,  correlation  results 
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only  prove  a  relationship  between  the  two  constructs.  This  insight  may  obscure  as  much 
as  it  reveals. 

Significance  of  Research 

Any  discussion  of  the  significance  of  this  study  must  start  with  acknowledging  the 
small  sample  size.  The  low  number  of  survey  responses  combined  with  the  homogenous 
pool  of  respondents  will  obviously  limit  how  far  these  results  can  be  generalized.  Even 
though  the  findings  are  interesting  these  factors  prevent  this  study  from  providing  truly 
significant  insight. 

Another  limiting  factor  to  declaring  this  study  significant  is  the  methods 
employed.  The  use  of  a  survey  for  this  effort  had  the  promise  of  providing  more  insight 
into  the  actual  practices  in  use.  However,  the  low  number  of  survey  responses  prevented 
the  use  of  more  rigorous  methods.  Correlation  is  useful  for  providing  insight  into  a 
relationship  between  two  constructs  exists  but  it  fails  to  prove  causation.  Recall,  also, 
that  the  correlation  methods  used  in  combination  with  the  low  sample  size  may  have 
produced  spurious  results  for  the  binary  questions  asked.  Additionally,  the  use  of 
correlation  may  allow  for  the  possibility  that  the  results  obtained  are  actually  from  a 
confounding  variable  that  exist  between  the  two  constructs  involved. 

Since  the  survey  responses  are  self-reported  without  validation,  additional 
limitations  remain.  For  one,  the  individuals  responding  may  not  have  complete  or 
accurate  information.  Additionally,  the  dependent  project  success  outcomes  were  not 
only  self-reported  but  self-reported  by  acquisition  personnel  that  may  be  closer  to  the 
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developer  than  to  the  user  or  customer.  This  is  especially  true  for  the  customer 
satisfaction  success  variable. 

It’s  important  to  repeat  the  point  made  earlier  that  all  of  this  is  transpiring  within 
the  minds  of  the  respondents.  There  is  no  tie  to  objective  measures  for  either  the  use  of 
the  risk  practices  or  to  the  project  success  outcomes  (cost,  schedule,  technical 
performance,  and  customer  satisfaction).  In  some  cases,  this  lack  of  an  objective  tie  may 
improve  accuracy  in  that  program  baselines  often  change  over  time  making  it  difficult  to 
tie  a  specific  practice  to  a  particular  outcome.  However,  this  lack  of  objective  validation 
poses  an  obvious  limitation  to  the  significance  of  the  findings. 

Still,  with  all  of  these  limitations,  this  study  does  provide  some  interesting 
findings.  It  finds  some  evidence  to  support  for  the  broad  practice  of  risk  management  as 
well  as  for  individual  practices.  It  also  provides  some  support  for  a  variety  of  policies 
and  prescriptions  dedicated  to  risk  management  and  project  success.  These  findings 
provide  an  answer  to  the  over-arching  question  that  motivated  this  study;  that  a  great  deal 
of  time  and  money  are  spent  on  risk  management  when,  previously,  its  relationship  and 
effectiveness  were  largely  unknown.  In  short,  this  study  provides  interesting  and  useful 
insight  that  can  aid  policy  makers,  project  management  practitioners,  and  educators  of 
risk  management. 

Recommendations  for  Action 

Given  the  findings  of  this  study,  a  number  of  recommendations  can  be  made.  For 
one,  policy  makers  have  some  added  insight  to  the  positive  relationship  risk  management 
has  with  project  success.  This  information  will  also  be  useful  to  project  management 
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practitioners  and  acquisition  personnel.  Finally,  edueators  of  risk  management  ean 
include  these  findings  in  their  treatment  of  risk  management. 

This  study  provides  some  useful  feedback  for  risk  management  policy  makers. 

The  findings  provide  useful  feedbaek  that  the  existing  policies  have  some  value.  Some  of 
the  findings  indieate  that  having  personnel  committed  to  risk  management  makes  it  more 
effeetive.  While  this  may  seem  obvious,  much  of  the  eurrent  projeet,  systems 
engineering,  and  aequisition  policy  focuses  more  on  the  risk  management  steps 
themselves  than  on  the  effort  behind  the  steps.  If  policy  makers  were  to  provide  this 
feedback  to  the  praetitioners,  it  may  help  them  see  the  value  which  might  lead  to  the 
desired  impaet:  more  suceessful  projeet. 

The  results  from  this  study  might  also  better  inform  praetitioners.  Since 
individual  practices  have  been  found  to  have  a  signifieant  and  positive  relationship  with 
projeet  sueeess,  they  might  be  more  likely  to  make  use  of  those  praetiees.  While  there’s 
no  elear  finding  of  causation  between  these  findings  and  project  success,  the  finding  of 
the  regression  models  indieate  that  an  organization’s  belief  in  the  effeetiveness  of  risk 
management  may  lead  to  the  desired  projeet  sueeess. 

Finally,  these  findings  may  aid  educators  of  risk  management.  So  much  of  risk 
management  education  relies  on  teaching  the  individual  risk  management  steps  (i.e., 
identifieation,  analysis,  mitigation,  etc.).  The  findings  of  this  study  provide  some 
validation  for  that  approach  but  also  highlight  the  importance  of  influenee  of  pereeption. 
Edueators  of  risk  management  should  also  relay  the  motivational  and  organizational 
behavior  aspect  of  risk  management.  The  steps  themselves  may  be  necessary,  but  they 
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are  insufficient.  The  students  should  understand  and  believe  the  value  of  what  they’re 
being  taught. 

Recommendations  for  Future  Research 

There  clearly  appears  to  be  value  in  performing  risk  management  in  an  effort  to 
guide  projects  to  meeting  their  success  targets.  However,  more  work  is  required  to  fiilly 
define  this  relationship.  This  study  showed  a  conclusive  relationship  between  risk 
management  and  project  success.  However,  future  studies  should  move  past  correlation 
to  better  define  this  relationship. 

This  study  provides  some  insight  but  suffers  from  limitations.  Primarily,  this 
study  provided  some  validation  for  a  similar  survey  delivered  to  industry  participants. 
Since  both  studies  found  similar  results,  it  certainly  appears  that  there  is  some 
relationship  worth  exploring  here.  A  future  study  might  seek  to  perform  a  similar 
analysis  but  use  more  objective  and  quantitative  data  to  evaluate  the  relationship  risk 
management  has  with  project  success  and  its  effectiveness. 

Risk  management  plays  a  significant  role  in  the  prescriptions  of  both  PMI  and 
INCOSE.  This  indicates  that  risk  management  is  an  important  role  in  both  project 
management  and  systems  engineering.  This  study  made  no  attempt  to  hold  constant  for 
the  other  PM  and  SE  practices.  Future  work  must  hold  constant  for  these  practices  as 
well  to  ensure  that  the  risk  management  findings  for  this  study  have  not  been  conflated 
with  those  other  two  sets  of  tools  to  aid  project  success. 

A  specific  area  of  further  research  would  be  to  expand  on  the  relationship 
between  organizational  behavior  and  risk  management.  Many  of  the  significant  questions 
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and  the  findings  from  the  regression  models  indieate  that  organizational  behavior  faetor 
influenee  risk  management.  This  is  an  area  that  appears  to  be  largely  ignored  by  the 
frameworks  reviewed  yet  is  seems  it  may  play  a  critical  factor  in  achieving  project 
success. 

With  the  correlation  findings  from  this  study,  future  research  should  be  free  to 
pursue  more  rigorous  methods  to  define  this  relationship.  Originally,  this  study  was  to 
rely  much  more  heavily  on  regression  models  to  define  the  risk  management  -  project 
success  relationship.  Low  survey  responses  prevented  the  use  of  more  rigorous  methods 
in  this  study.  Future  studies  might  be  better  served  by  avoiding  survey  methods  and 
employ  more  objective  methods  that  rely  less  on  survey  work. 

Risk  management  certainly  appears  to  have  a  relationship  with  project  success 
with  respect  to  cost,  schedule,  technical  performance,  and  customer  satisfaction  outcomes 
for  this  sample  population.  Numerous,  specific  risk  management  practices  are  found  to 
correlate  significantly  with  these  project  success  measures.  However,  its  exact 
relationship  and  its  effectiveness  remain  less  clear.  This  study  finds  some  support  for  a 
number  of  risk  management  frameworks  and  specific  risk  practices  but  more  work 
remains  to  be  able  to  generalize  these  findings  to  a  broader  population. 
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Appendix  A:  Survey 


DoD  Risk  Management  Practices 


AN  AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  SURVEY 


SCN:  2013-05 


Privacy  Notice 


The  following  information  is  provided  as  required  by  the  Privacy  Act  of  1974: 

Purpose:  The  goal  of  the  survey  is  to  understand  betterwhat  the  current  state  of  practice  in  industry  and  government  services  is  regarding  the 
management  of  risk  in  development  programs  and  projects.  Results  from  this  survey  will  be  used  to  indentify  areas  for  improvement  in  the  areas 
of  policy  and  education/training.  The  survey  was  developed  by  MIT's  Lean  Advancement  Initiative  (LAI)  in  collaboration  with  the  Air  Force  Institute 
of  Technology  and  Futron. 

Participation:  Participation  is  strictly  VOLUNTARY.  You  are  not  required  to  participate  in  this  survey. 

Confidentiality:  We  would  greatly  appreciate  your  participation.  ALL  ANSWERS  WILL  BE  KEPT  STRICTLY  CONFIDENTIAL. 


Instructions 


During  this  survey,  we  will  ask  questions  regarding  risks,  risk  management  practices  and  the  success  of  risk  management  in  projects  and 
programs. 

For  the  purpose  of  this  survey,  we  follow  the  definitions  and  guidance  of  the  ISO  31000  standard.  Risk  is  defined  as  the  effect  of  uncertainties  on 
objectives;  risk  management  is  defined  as  coordinated  activities  to  direct  and  control  an  organization  with  regard  to  risk.  When  you  answer  the 
questions,  we  ask  you  to  observe  the  following  rules: 

•  Please  pick  one  program  or  project  to  use  as  a  reference  when  answering  the  questions. 

•  Always  use  this  one  program/project  as  a  reference  for  all  questions. 

•  Please  choose  a  program/project  with  a  focus  on  development  (not  only  production). 

•  Please  choose  the  development  program/project  that  was  finished  most  recently,  if  possible  within  the  last  6  months. 

The  survey  will  ask  questions  based  on  the  following  structure 

1 .  General  Questions  -  Organization 

2.  General  Questions  -  Program/Project 

3.  Risk  Management  Processes 

4.  Risk  Management  Performance 

The  Risk  Management  Processes  section  will  be  further  decomposed  by: 

1.  Planning  and  Preparation 

2.  Risks  and  their  Impact 

3.  Risk  Analysis 

4.  Risk  Evaluation 

5.  Risk  Mitigation 

6.  Risk  Monitoring 

If  you  cannot  or  wish  not  to  answer  a  question,  please  leave  that  answer  blank. 

The  survey  should  take  25-30  minutes. 
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General  Questions  on  Your  Organization 

Please  provide  some  information  about  your  organization  and  yourself. 


1. 

What  type  of  organization  do 
you  work  for? 

t  E 

2. 

What  is  the  yearly  budget  of 
your  company  or  government 
equivalent  (for  government 
employees,  consider  the 
budget  of  your  PEO) 

r 

3. 

What  describes  your  role 
best  during  the  program  or 
project? 

I  S 

Yes  No 

1  2 

5. 

Did  your  project  allocate  a 
significant  portion  (at  least 

1 0%  of  yearly  budget)  to 
conduct  risk  management 
activities? 

6. 

If  government,  which  service 
do  you  work  for? 

I  □ 

7. 

If  you  selected  'other'  above, 
please  specify  here 

Contractor  (working 

Active  Duty  Civilian  directly  in  a  government 

program  office) 

1  2  3 

8. 

If  government,  are  you  active 
duty  or  civilian? 

9. 

How  many  years  of 
acquisition  experience  do  you 
have? 

• 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide  additional 
information,  please  enter  it  in  the  box  below: 
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General  Questions  on  Your  Program/Project  (1/2) 

Please  provide  some  general  information  on  the  program/project  you  chose  as  the  example  for  this  survey 


$500k-  $1m-  $5m-  $10m-  $50m-  $500m  do  not 

$500k  $500m  $1bn 

123456789  10 

11. 

Development  budget  for  all 
contractors/suppliers  for  the 
program/project 

12. 

Development  budget  within  your 
organization  for  program/project 

13. 

What  types  of  industry  sector  does  the 
program  fit  best? 

16.  What  risk  management  models  w 
relevant  for  the  design  of  your  risi 
management  process? 


|~~|  Department  of  Defense  Risk  Management  Guide  for  DoD  Acquisition 
I — [  Air  Force  Life  Cycle  Risk  Management  Process 

|— I  Project  Management  Institute  (PMI)  project  risk  management  process  (part  of  the 
i— i  Project  Management  Body  of  Knowledge 

I — [  Risk  management  process  of  PRINCE2  project  management  framework 

□  NASA  Risk-informed  Decision  Making  (RIDM)or  Continuous  Risk  Management 
(CRM)  process 

□  INCOSE  risk  management  process  from  the  Systems  Engineering  Handbook 
□  ISO  31000  standard  "Risk  management-  principies  and  guideiines" 

□  Do  not  know 
[~|  Other 


17. 

What  development  approaches  or 
philosophiesplayed  a  significant  role  in 

11 

Waterfall  (e.g.  Stage  Gate,  V-Model,  DoD  5000) 

□ 

Spriral  development 

□ 

|Agile  development  | 

□ 

Design  for  Six  Sigma 

□ 

|Lean  Product  Development 

□ 

I^Snot  Know 

□ 

Other 

Prooram 

System 

Component 

fCoordhnation 

Integrator 

supplier/teir 

supplier/tier 

Lower-tier 

of  entire 

(Organization 

-1  supplier 

-2  supplier 

supplier/tier- 

develooment 

mainly 

(Main 

(Supplier  for 

3  or  lower 

effort 

reponsible 

supplier  of 

key 

(Supplier 

for  the 

a  high-level 

components 

that  delivers 

customTr 

customer  or 

system, 

for  a 

parts  for 

contractors 

contractor 

integrator 

specific 

system 

and 

side) 

of  that 

system  or 

components) 

suppliers)) 

system) 

assembly) 

1 

2 

3 

4 

5  6 

18. 

At  what  level  of  the  program/project 

enterprise  are  you  working? 

Optionai:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  wouid  iike  to  provide  additionai  information,  please 
enter  it  in  the  box  beiow: 
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General  Questions  on  Your  Program/Project  (2/2) 

The  following  questions  will  ask  you  to  generally  characterize  the  project/program  posed  in  the  5  areas  of 

•  Technology 

•  Customer 

•  Company 

•  Supplier 

•  Market 

Regarding 

•  novelty  and 

•  complexity 


Please  rate  the  challenge  that  the  program/project  posed  for  your  organization  regarding  technology: 


Very  low  Low  Average  High  Very  high 

1  2  3  4  5 

20. 

Technology  experience:  Familiarity  of  your 
organization  with  key  technologies. 

21. 

Technology  Complexity:  Size  and  level  of 
integration  of  the  technical  system  (mechanical, 
electronics  and  software) 

Please  rate  the  challenge  that  the  program/p  reject  posed  for  your  organization  regarding  the  customer: 


Very  low  Low  Average  High  Very  high 

1  2  3  4  5 

22. 

Experience  with  customer  or  stakeholders: 
Familitarity  of  your  organization  with  key 
customers  and  stakeholders. 

23. 

Customer  or  stakeholder  complexity:  Number  and 
diversity  of  customers  or  stakeholders. 

Please  rate  the  challenge  that  the  program/p  reject  posed  for  your  organization  regarding  the  internal  processes  and  skills: 


Very  low  Low  Average  High  Very  high 

1  2  3  4  5 

24. 

Experience  with  relevant  processes  and  skills: 
Familiarity  of  your  organization  with  the  relevant 
processes  and  skills  needed  to  execute  the 
project/program. 

25. 

Complexity  of  relevant  processes  and  skills: 
Number,  difficulty  and  variety  of  processes  and 
skills  needed  in  your  organization  to  execute  the 
project/program. 

Please  rate  the  challenge  that  the  program/project  posed  for  your  organization  regarding  the  supply  chain: 


Very  low  Low  Average  High  Very  high 

1  2  3  4  5 

26. 

Experience  with  supply  chain:  Familiarity  of  your 
organization  with  the  supply  chain  needed  to 
execute  the  project/program. 

27. 

Complexity  of  supply  chain:  Size,  diversity  and 
level  of  integration  of  the  project's  or  program's 
supply  chain. 

Please  rate  the  challenge  that  the  program/p  reject  posed  for  your  organization  regarding  external  factors: 


Very  low  Low  Average  High  Very  high 

1  2  3  4  5 

28. 

Experience  with  external  factors:  Familiarity  of 
your  organization  with  the  external  factors  (e.g. 
Other  organizations,  legal  and  regulatory 
environment). 

29. 

Complexity  of  external  factors:  Number  and 
diversity  of  external  factors  (e.g.  other 
organizations,  legal  and  regulatory  environment). 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide  additional  information,  please 
enter  it  in  the  box  below. 


30. 
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Risk  Management  Process  -  Planning  and  Preparation 

Integration  of  stakeholders  in  communication  and  consultation  of  risk  management  activities.  Choosing  the  right  processes,  tools  and 
methods  for  risk  management. 


Please  indicate  your  assessment  of  the  way  risk  management  was  executed. 


Strongly  Disagree  Neither  Agree  ^  Strongly 

Disagree  or  Disagree  Agree 

1  2  3  4  5 

31. 

Our  employees  are  motivated  to 
perform/implement  risk  management. 

32. 

Our  risk  management  has  available, 
qualified  experts  to  help  implement  the 
processes. 

33. 

There  are  available  resources  or  manpower 
to  conduct  risk  management. 

34. 

Our  risk  management  explicitly  addresses 
uncertainty. 

35. 

Our  risk  management  is  systematic, 
structured  and  timely. 

36. 

Our  risk  management  is  based  on  the  best 
available  information. 

37. 

Our  risk  management  is  tailored  to  specific 
program/project  needs. 

38. 

Our  risk  management  takes  human  and 
cultural  factors  into  account. 

39. 

Our  risk  management  is  transparent  and 
inclusive  towards  all  stakeholders. 

40. 

Our  risk  management  is  dynamic,  iterative 
and  responsive  to  change. 

Please  indicate  which  of  the  following  statements  regarding  stakeholder  communication  and  consultation  apply  to  your  risk 
management. 


_  There  is  a  formal  document  (e.g.  risk  management  plan)  that  defines 

I — I  when,  how  and  by  whom  the  risk  management  process  is  executed 
I — I  There  is  a  board  that  oversees  risk  management  activities  of  the 
' — '  program/project 

„  Risks  and  risk  management  activities  are  communicated  to  stakeholders 
' — '  (including  management) 

„  Risks  are  communicated  as  consolidated  reports  (e.g.  PDF  files  are  email 
attachments) 

Risks  are  communicated  via  managed  risk  register/database 


Please  indicate  if  the  following  statements  apply  to  the  risk  management  process  step  in  your  project/program. 


Stronolv  r,.  Neither  Aaree  .  Stronolv 

Disagree  nor  Disagree  Agree 

1  2  3  4  5 

42. 

We  tailor  the  risk  management  process  and 
the  methods  to  the  specific 
program/project. 

43. 

We  coordinate  and  integrate  risk 
management  activities  of  different  functions 
and  across  the  hierarchy. 

C' 

44. 

Risk  management  is  integrated  with  higher- 
level  risk  management  processes,  e.g. 
portfolio-level  risk  management  or 
enterprise-level  risk  management. 

•I' 

45. 

The  risk  management  process  is  effectively 
integrated  with  other  project/program 
management  processes. 

46. 

Risk  management  teams  are  cross 
functional  and  cross-organizational. 

- 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide  additional  information, 
please  enter  it  in  the  box  below. 
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Risk  Management  Process  -  Types  of  risk  and  their  impact 

In  the  following,  a  list  of  risks  is  presented.  Please  indicate  below  the  impact  that  these  risks  did  have  on  your  program/projeot.  The  impact  can  be  on  and  program  or 
project  target,  e.g.  cost,  schedule  or  performance. 

Please  rate  in  the  following  questions  the  overall  risk  impact  in  the  project: 

•  Not  occurred:  The  described  risk  did  not  play  a  significant  role  in  the  program/project. 

•  Very  low  impact:  The  risk  occurred,  but  could  be  dealt  with  in  the  routine  workflow. 

•  Medium  impact:  The  risk  required  special  attention  and  resource  allocation  to  overcome. 

•  Very  high  impact:  The  risk  significantly  threatened  the  overall  program/project  success. 

If  you  don't  know  the  answer,  please  leave  the  question  blank. 


Risks  regarding  organizational  efficiency. 
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Risk  Management  Process  -  Risk  Analysis  and  Quantification 
Clarification  of  risks  with  sufficient  accuracy 


78. 

Please  indicate 
what  dimensions 

Cost 

were  used  to 
quantify  the 

Technical  performance  or  quality 

impact  of  risks: 

Human  health,  environmental,  systems  safety  or  reliability 

Schedule 

Supportability(infrastructure,  logistics,  workforce) 

General  customer  utility  or  customer  satisfaction 

Other 

Please  indicate  how  often  the  different  methods  were  used  to  quantify  risks. 


Never  Rarely  used  Sometimes  often  used  Always  used 

used 

1  2  3  4  5 

79. 

No  direct 
quantification,  but 
rank  ordering  of 
risks,  e.g.  1  to  10 
for  top  10  risks. 

80. 

Assessment  of 
risks  on  scales, 
e.g.  1-5  scale  for 
probability  and 
impact. 

81. 

Probabalistic  Risk 
Assessment 
(PRA)  method. 

82. 

Probability 
distributions,  e.g. 
triangular 
distributions  with 
minimum,  most 
likely  and 
maximum. 

83. 

Monte  Carlo 
simulations  (or 
similar)  to 
aggregate 
different  types  of 
risk  estimates. 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide 
additional  information,  please  enter  it  in  the  box  below. 


CONTINUE 
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Risk  Management  Process  -  Risk  Evaluation 
Prioritization  of  risks  for  for  proper  treatment 


How  often  did  you  use  the  following  techniques  to  make  decisions  about  risks  in  your  project/program? 


Never  Rarely  used  Sometimes  often  used  Always  used 

used 

1  2  3  4  5 

85. 

Make  go/no-go 
decisions  based 
on  risk 
assessment. 

86. 

Resources  are 
allocated  to 
reduce  target 
risks  as  early  as 
possible. 

87. 

Risk  assessments 
are  used  to  set 
more  'realistic'  or 
'achievable' 
objectives. 

88. 

Forecasts  and 
projections  (e.g. 
cost,  schedule, 
performance)  are 
adjusted  based 
on  risk 

assessments. 

89. 

The  results  of  the 
risk  analysis  are 
considered  in 
making  technical, 
schedule  and/or 
cost  trade-offs. 

90. 

Decisions  are 
made  based  on 
risk-benefit  trade¬ 
offs,  e.g.  larger 
risks  are  only 
acceptable  for 
significant 
expected 
benefits. 

91. 

Risk-benefit  trade 
-offs  are  used 
systematically  to 
favor  'low  risk- 
high  benefit' 
options  and 
eliminate  'high 
risk-low  reward' 
options. 

92. 

Contracts  are 
derived  from 
detailed  cost  risk 
assessments. 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide 
additional  information,  please  enter  it  in  the  box  below. 


CONTINUE 
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RMk  Management  Procaee  -  Riek  MMgatkMi 
Treatment  or  neks  wttheniBCOvemmgalan  actkxK 


94. 

Please  Indlcaie  which  dimensions  are  used  to  evakiale  nsk  mitgalion  actions  betore 
they  are  implemented. 

Coettreeource  needs  for  mitigation  action 

Time  requirement  for  mitigation  action 

Reduction  of  Imfuct  or  nsk  through  mitigation  action 

Reduction  of  probaMllty  of  occunence  or  nak  through  mitigation  action 

Other 

Please  rate  m  an  Miowlng  questions  on  tie  overall  nsk  reduction  achieved  Oy  different  mttiganon  actions  (e.g.  Dy  reducing  prooaMlty  or  occurrence  or  reducing  the  Impact  or  risks): 

•  Very  low  nek  reduction:  The  mMgatkin  action  allgritly  reduced  a  al^incant  nek. 

•  Low  nsk  reduction:  The  mitigation  action  reduced  a  algnltlcaiit  nek. 

•  Medium  nek  reduction:  The  mtlgatlon  action  reduced  a  numlier  ot  algnlflcant  naka. 

•  High  nsk  reduction:  The  mitigation  action  reeolved  one  algnltlcant  nak. 

•  Vary  Mgh  nak  reduction:  The  mitigation  action  raeolvod  several  algnlflcant  rtska. 

IT  you  don't  know  the  answer,  pteaae  leave  the  question  blank. 


Mitigation  actions  to  reduce  naka  regarding  organizational  efflclency: 


vsrylowrlek  Low  risk  Mediwnrtak  High  risk  very  Mgh  risk 

Not  used  reduction  reduction  reduction  reduction  reduction 

achieved  achieved  acMaved  achieved  achieved 

1  2  3  4  5  6 

95. 

Suppiier>enterpnse  titegratlon  and  management  e.g.  process  harmonlzacion  and  data 
Integration. 

96. 

ContracMTigovemment  integration,  e.g.  reporting,  teedback.  etc. 

97. 

Organzatlon-intemai  integration,  eg.  process  harmonization  and  data  iniegralan. 

96. 

Financial  reserves. 

99. 

schedule  reserves. 

too. 

Contractual  shanng  or  cost  overruns  with  conlractor(s). 

161. 

Contractual  shaitig  or  cost  overruns  with  other  oiganlzatioris. 

162. 

Fkm  Rxed  Prtce  contracts. 

163. 

Hoidhg  excess  resources  (e.g.  manpower.  Inventory  or  tacHfles). 

164. 

Other. 

16S. 

tr  you  selected  'other  above,  please  specify  here 
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VerylDwrtfllc  LDWitak  Median  rtalr  tUghitel!  Very  High  nsJi: 

not  uaed  reductton  reduction  nedudUiHi  reductlan  reductioi 

actil'Bved  adiloved  achieved  arXiloved  achlaved 

1  2  3  4  5  e 

1«. 

Derailed  cost,  achedulaand  peiftinnana'aJniulaaoria^  liade-DlTsEiH^ 

107. 

Setf-asseEsmente. cmUniBits  hiprowement and  InnptementaflonoftKEt  pracdDce ije.g. 
Six:  Sigma.  KaizEn). 

1W. 

MDre  detalod  diealgn  rnlewa,  Incfdaaad  prQce€«  monEwlng. 

1®. 

Tralrlr^  program  or  EpeDlall^  Dars€rpadiiolncrBa6a£kll  levEl. 

111. 

DEflnoisEandard  warf  orislanilara  priKSBaes' to  Incfflease  praoesE  reHatHlEy. 

111. 

Improved  engineering  change  pmcesE  to  Epeed  ifi  changsE. 

112. 

Adaptatfcxi  Df  PD  prooe^seE  Id  match  Epedflc  pmfecE  regtdrernentE. 

113. 

Aeave  Ihtem^  loMiylng  towa^  top  management  tD  promote  project'[HogaTi. 

11*. 

Other. 

IIS. 

IT  you  Eelected  Wrer'  above,  pte^  Epectty  here 

MEbgatlon  amona  Ed  rDduct!  rtsK^a  raganlng  rsqulrsnicntB.  coniradlro  and  CDmjallanca: 


VDiytDwrteh  LDWilaK  Median  rtelc  HlghiteK  Very  High  dsk: 

HDt  ufled  reducdiHi  reduction  redudlDii  reduction  reducHm 

sdileved  ecHleved  achieved  aiXileved  ler^leved 

1  2  3  *  5  e 

lie. 

Help  DLKtmter  (UEerl  underetand  mfiat  thet  needs  are  and  nnahe  Irade-onife  (e.g.  MATE 
or  oUw  ^ade-ol?  ElmutaPonE  or  c^culaUons)-. 

117. 

Management  (andlre-negotlallon.  irneceEsary'l  or  requirements. 

iia. 

Acdve  toDPy Ing  with  key  Elakehctdem  outElde  cl'  dlred  ciEtomendonCjactor  neiatlonEllp. 
e.gi  legiiaiorY'  or  policy  makerE. 

119. 

klonlfiir  aifflvIBeE  of  oOier  crgamzadons  (e.g.  technalogy  daXosureE.  Wddng  strategy, 
product  launcheE,  market  mMes..  aialy^deilstlig  prodictE,  etc.) 

12ffi. 

Other. 

121. 

tr  you  Eelected  'Wier  ^love.  ptea&e  Epectiy'  here 

Uttgatlon  acdona  Ed  r»duMi  EDChnologlcal  rteka: 


VDiy  low  flak  LDHrilak  Median  rtelc  Hlghitek  Very  High  risk 

HDt  usad  reducdiHi  reduction  mducdiHi  reductlan  redu^«i 

sidilBVBd  acHleved  achieved  achlavDd  achlevHl 

1  2  3  4  5  8 

122. 

Inote^ed  tectlng  and  protolYplng  acttvtIleE. 

123. 

Reuse  dexlEElig  oomponmte  or  oTT-tbe-Eheir  componenlE. 

124. 

Deretop  Had  We  product  arctittecture  {e.gL  modUar  ptaitmmi}. 

125. 

SliftX  DonllguraElon  control  fei  nnanage  and  mirimtze  compteilty  and  incertalinty. 

12fi. 

Engineering  wEH  redundancy  dr  EaTtety  margUiE. 

127. 

Pursue  Eevem  englneeilng  eWutlonE  In  paraPel  ije.g.  Eet-bsEed  design]; 

128. 

Focus  on  destgi  lor  manuliacSjrlttg  andtif  design  Ibr  servloe. 

129. 

Other. 

13fl. 

IT  you  selected  Wier^iove.  ptease  spectty  Here 

Dpthtnsl:  r]fa«j  have  any  HmnneniB  ragamlng  EliaqucBilDita  on  itila  paga  Dr  If  you  would  like  Ed  provide  adiltiDnal  InfDnratiDn,  pleaaD  antarltln  Eha  box  below. 
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Risk  Management  Process  -  Monitoring  and  Review 

Sufficient  monitoring  of  risks  and  execution  of  the  risk  management  process. 


To  what  degree  do  you  agree  or  disagree  to  the  following  statements  on  Monitoring  and  Review  processes? 


Completely  Disaoree  Neither  Agree  ^  Completely 

Disagree  ^  nor  Disagree  ^  Agree 

1  2  3  4  5 

132. 

Risks  were  escalated  to  senior 
management  according  to  guidelines. 

133. 

Risks  were  regularly  re-assessed 
according  to  guidelines,  e.g.  after  specific 
events  or  after  a  certain  time  interval. 

134. 

The  risk  management  process  was 
regularly  reviewed  and  Improved. 

135. 

A  formal  feedback  system  was  used  to 
monitor  the  execution  of  risk  mitigation 
actions. 

136. 

An  early  warning  system  was  used  to 
track  critical  risks  and  decide  on  activating 
mitigation  measures. 

How  often  are  the  following  elements  formally  reviewed  In  your  organization? 


Once  Only 

Daily  Weekly  Monthly  Quarterly  Annually  Never 

annually  program  specific 

start)  events 

123  4  5  6  789 

137. 

Identification  of  new  risks. 

138. 

Quantification  of  risks. 

139. 

Risk  management  measures. 

140. 

Risk  management  process. 

1 

141. 

Based  on  occurrence  of  specific  events. 

t- 

142. 

If  based  on  specific  events,  please 
specify: 

143. 

Please  Indicate  If  the  following  methods 
are  used  for  monitoring: 

1—1  Risk  register  or  risk  catalog 

Top  10  risks 

Risk  elimination  or  risk  burn-down  plans 

Risk  mitigation  plans 

Graphical  risk  metrics  dashboard 

144. 

Please  Indicate  If  the  following  Key 
Performance  Indicators  are  used  to  track 
risks: 

Tracking  of  error/issue/failure  rates 

Tracking  of  number  of  total  risks 

Tracking  of  number  of  retired  risks 

Tracking  of  aggregated  risk  severity 

Tracking  of  number  of  risk  mitigation  measures 

|— 1  Tracking  of  resource  expenditure  on  risk  mitigation  measures  (cost, 

' — '  manpower). 

Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like  to  provide  additional  information, 
please  enter  It  In  the  box  below. 
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Risk  Managemeht  Performance - 

Questions  to  assess  how  effectively  the  program  dealt  with  risk  and  uncertainty  and  how  stable  it  ran. 


Please  indicate  to  what  extent  you  agree  with  the  following  statements  regarding  the 
role  and  perception  of  risk  management  in  the  program/project: 


Please  indicate  to  what  extent  you  agree  with  the  following  statements  regarding  the  influence  of  risk  management  on  the  program/project: 


How  strongly  do  the  following  statements  apply  to  the  overall  program/project  execution? 


Optional:  If  you  have  any  comments  regarding  the  questions  on  this  page  or  if  you  would  like 
to  provide  additional  information,  please  enter  it  in  the  box  below. 
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Thank  you  for  your  time  and  thoughtfulness  filling  out  and  submitting  this  survey.  Your  thoughts  are 
greatly  appreciated. 


173. 


General 
feedback: 
If  you 
have  any 
general 
comments 


regarding 
the  survey 
(too  long 
or  too 
short,  too 
much  or 
too  little 
detail, 
etc.), 
please  let 
us  know 
here: 
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Abbreviations 


APDP 

Acquisition  Professional  Development  Program 

CMMl 

Capability  Maturity  Model  -  Integrated 

CPFF 

Cost  Plus  Fixed  Fee 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

GAO 

General  Accountability  Office 

lEC 

International  Electrotechnial  Commission 

INCOSE 

International  Council  on  Systems  Engineering 

1PM 

Intermediate  Project  Management  Skills  Course 

ISO 

International  Organization  for  Standardization 

JSF 

Joint  Strike  Fighter 

LFP 

Lightweight  Fighter  Program 

MDAP 

Major  Defense  Acquisition  Program 

NASA 

National  Aeronautics  and  Space  Administration 

PM 

Program  Management,  Project  Management 

PMl 

Project  Management  Institute 

PMT 

Program  Management  Tools 

RM 

Risk  Management 

SE 

Systems  Engineering 

TRL 

Technology  Readiness  Level 
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